Need help on simple linear regression proof

**sabercn** · September 15th 2012, 03:37 PM

I want to derive the least square estimation of the coefficients for y=B_0+B_1*x_1+e

Can someone walk me through how to get from B to D in the image below?

http://i.imgur.com/9zwvO.jpg

I can do it through taking the derivative of the residual sum of squares, but I want to know how to do the proof through the general formula.

**Prove It** · September 15th 2012, 09:26 PM

I'm going to call $\displaystyle \begin{align*} \mathbf{Y} = \left[ \begin{matrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{matrix} \right], \mathbf{X} = \left[ \begin{matrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{matrix} \right] \end{align*}$ and $\displaystyle \begin{align*} \mathbf{\beta} = \left[ \begin{matrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_n \end{matrix} \right] \end{align*}$ , then, assuming no error, we have the equation $\displaystyle \begin{align*} \mathbf{Y} &= \mathbf{X}\mathbf{\beta} \end{align*}$ , with the aim to solve for $\displaystyle \begin{align*} \mathbf{\beta} \end{align*}$ .

Normally, what could be done is to premultiply both sides by $\displaystyle \begin{align*} \mathbf{X}^{-1} \end{align*}$ , but we can't do that in this case because $\displaystyle \begin{align*} \mathbf{X} \end{align*}$ is not a square matrix. But we can CREATE a square matrix by premultiplying both sides by $\displaystyle \begin{align*} \mathbf{X}^T \end{align*}$ first. This gives the equation $\displaystyle \begin{align*} \mathbf{X}^T\mathbf{Y} = \mathbf{X}^T\mathbf{X}\mathbf{\beta} \end{align*}$ .

Now that $\displaystyle \begin{align*} \mathbf{X}^T\mathbf{X} \end{align*}$ is a square matrix, to solve this equation for $\displaystyle \begin{align*} \mathbf{\beta} \end{align*}$ we need to premultiply both sides by $\displaystyle \begin{align*} \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \end{align*}$ , giving

$\displaystyle \begin{align*} \left( \mathbf{X}^T\mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T\mathbf{X} \mathbf{\beta} \\ \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \mathbf{I} \mathbf{\beta} \\ \left( \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T \mathbf{Y} &= \mathbf{\beta} \end{align*}$

Did this help?

**sabercn** · September 15th 2012, 09:42 PM

Edit. Thanks for the help, I solved the proof.

Thread: Need help on simple linear regression proof

LinkBack

Thread Tools

Display

Need help on simple linear regression proof

Re: Need help on simple linear regression proof

Re: Need help on simple linear regression proof

Similar Math Help Forum Discussions

finding the p-value in simple linear regression

Simple Linear Regression

Regression fundamentals (linear, non linear, simple, multiple)

SSE simple linear regression

Simple Linear Regression

Search Tags