# Need help on simple linear regression proof

• Sep 15th 2012, 03:37 PM
sabercn
Need help on simple linear regression proof
I want to derive the least square estimation of the coefficients for y=B_0+B_1*x_1+e

Can someone walk me through how to get from B to D in the image below?

http://i.imgur.com/9zwvO.jpg

I can do it through taking the derivative of the residual sum of squares, but I want to know how to do the proof through the general formula.
• Sep 15th 2012, 09:26 PM
Prove It
Re: Need help on simple linear regression proof
I'm going to call \displaystyle \begin{align*} \mathbf{Y} = \left[ \begin{matrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{matrix} \right], \mathbf{X} = \left[ \begin{matrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{matrix} \right] \end{align*} and \displaystyle \begin{align*} \mathbf{\beta} = \left[ \begin{matrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_n \end{matrix} \right] \end{align*}, then, assuming no error, we have the equation \displaystyle \begin{align*} \mathbf{Y} &= \mathbf{X}\mathbf{\beta} \end{align*}, with the aim to solve for \displaystyle \begin{align*} \mathbf{\beta} \end{align*}.

Normally, what could be done is to premultiply both sides by \displaystyle \begin{align*} \mathbf{X}^{-1} \end{align*}, but we can't do that in this case because \displaystyle \begin{align*} \mathbf{X} \end{align*} is not a square matrix. But we can CREATE a square matrix by premultiplying both sides by \displaystyle \begin{align*} \mathbf{X}^T \end{align*} first. This gives the equation \displaystyle \begin{align*} \mathbf{X}^T\mathbf{Y} = \mathbf{X}^T\mathbf{X}\mathbf{\beta} \end{align*}.

Now that \displaystyle \begin{align*} \mathbf{X}^T\mathbf{X} \end{align*} is a square matrix, to solve this equation for \displaystyle \begin{align*} \mathbf{\beta} \end{align*} we need to premultiply both sides by \displaystyle \begin{align*} \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \end{align*}, giving

\displaystyle \begin{align*} \left( \mathbf{X}^T\mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T\mathbf{X} \mathbf{\beta} \\ \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \mathbf{I} \mathbf{\beta} \\ \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \mathbf{\beta} \end{align*}

Did this help?
• Sep 15th 2012, 09:42 PM
sabercn
Re: Need help on simple linear regression proof
Edit. Thanks for the help, I solved the proof.