Self-designed objective for multiple linear regression

Jul 2013
17
0
Beijing
A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:


$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+$ $\epsilon_{i}$, $i=1,...,n$


I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example to maximize the Shannon entropy of the sequences of $\epsilon$ (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?


Thank you for your help.
 

chiro

MHF Helper
Sep 2012
6,608
1,263
Australia
Hey baiyang11.

Are you familiar with expectation maximization? Also you will need to be familiar with multi-variable optimization techniques and constructing an iteration system (of matrices) to converge to the true value under these conditions.

When looking at GLM's you will find that they use matrix calculus to iterate towards a solution (also in EM) and you can combine that style of thinking with optimization and iterate towards a solution.

The matrix approach will require a bit more linear algebra than what is normally touched on in statistics.
 
Jul 2013
17
0
Beijing
Thanks for your reply.

Where do you think I should start with to construct a way to model and solve this problem?
 

chiro

MHF Helper
Sep 2012
6,608
1,263
Australia
Look at GLM's and Expectation Maximization algorithms along with Optimization in many variables.

Can't really give more advice with the information you have given.