Template:Linear regression (least squares) gompz

Linear Regression (Least Squares)
The method of least squares requires that a straight line be fitted to a set of data points. If the regression is on $$Y$$, then the sum of the squares of the vertical deviations from the points to the line is minimized. If the regression is on $$X$$, the line is fitted to a set of data points such that the sum of the squares of the horizontal deviations from the points to the line is minimized. To illustrate the method, this section presents a regression on $$Y$$. Consider the linear model [2]:


 * $${{Y}_{i}}={{\widehat{\beta }}_{0}}+{{\widehat{\beta }}_{1}}{{X}_{i1}}+{{\widehat{\beta }}_{2}}{{X}_{i2}}+...+{{\widehat{\beta }}_{p}}{{X}_{ip}}$$

or in matrix form where bold letters indicate matrices:
 * $$Y=X\beta $$


 * where:


 * $$Y=\left[ \begin{matrix}

{{Y}_{1}} \\ {{Y}_{2}} \\ \vdots  \\ {{Y}_{N}} \\ \end{matrix} \right]$$


 * $$X=\left[ \begin{matrix}

1 & {{X}_{1,1}} & \cdots & {{X}_{1,p}}  \\ 1 & {{X}_{2,1}} & \cdots & {{X}_{2,p}}  \\ \vdots & \vdots  & \ddots  & \vdots   \\ 1 & {{X}_{N,1}} & \cdots & {{X}_{N,p}}  \\ \end{matrix} \right]$$
 * and:


 * $$\beta =\left[ \begin{matrix}

{{\beta }_{0}} \\ {{\beta }_{1}} \\ \vdots  \\ {{\beta }_{p}} \\ \end{matrix} \right]$$

The vector $$\beta $$  holds the values of the parameters. Now let $$\widehat{\beta }$$  be the estimates of these parameters, or the regression coefficients. The vector of estimated regression coefficients is denoted by:


 * $$\widehat{\beta }=\left[ \begin{matrix}

{{\widehat{\beta }}_{0}} \\ {{\widehat{\beta }}_{1}} \\ \vdots  \\ {{\widehat{\beta }}_{p}} \\ \end{matrix} \right]$$

Solving for $$\beta $$  in Eqn. (linear) requires the analyst to left multiply both sides by the transpose of $$X$$,  $${{X}^{T}}$$ :


 * $$({{X}^{T}}X)\widehat{\beta }={{X}^{T}}Y$$

Now the term $$({{X}^{T}}X)$$  becomes a square and invertible matrix. Then taking it to the other side of the equation gives:


 * $$\widehat{\beta }={{(}^{T}}^{-1}{{X}^{T}}Y$$