ReliaWiki - User contributions [en]

Multiple Linear Regression Analysis

2012-08-24T06:36:11Z

Pengying niu: /* Residual Analysis */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}) &= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}: {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) & = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}) &= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}:{{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})& = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}}) &= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
{{R}^{2}} & = & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
R_{adj}^{2} &= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
{{d}_{i}}&= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
{{r}_{i}} &= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:35:30Z

Pengying niu: /* Coefficient of Multiple Determination, {{R}^{2}} */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}) &= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}: {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) & = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}) &= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}:{{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})& = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}}) &= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
{{R}^{2}} & = & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
R_{adj}^{2} &= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:34:49Z

Pengying niu: /* Sequential Sum of Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}) &= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}: {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) & = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}) &= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}:{{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})& = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}}) &= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:33:21Z

Pengying niu: /* Partial Sum of Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}) &= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}: {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) & = & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
{{f}_{0}} &= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:31:51Z

Pengying niu: /* Partial Sum of Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}: {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:30:18Z

Pengying niu: /* Calculation of the Statistic {{F}_{0}} */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}: {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:28:42Z

Pengying niu: /* Test on Individual Regression Coefficients ( t Test) */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
p\text{ }value & = & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:27:38Z

Pengying niu: /* Test on Individual Regression Coefficients ( t Test) */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
{{{\hat{\sigma }}}^{2}} &= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
C &= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
se({{{\hat{\beta }}}_{1}}) &= & \sqrt{0.1557}=0.3946 \\
se({{{\hat{\beta }}}_{2}})& = & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}} &= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}} &= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
{{t}_{\alpha /2,n-(k+1)}} &= & {{t}_{0.05,14}}=1.761 \\
-{{t}_{\alpha /2,n-(k+1)}} & = & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:26:01Z

Pengying niu: /* Test for Significance of Regression */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}:& {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}:& {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
H & = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
S{{S}_{R}} & = & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
M{{S}_{R}}& = & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
S{{S}_{E}} &= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
M{{S}_{E}} &= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
{{f}_{0}}& = & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
{{f}_{\alpha ,k,n-(k+1)}} &= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:17:40Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
\hat{y}(47,31)& = & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:16:38Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
{{e}_{i}} & = & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
{{e}_{5}}& = & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
& \hat{y}(47,31)= & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:15:49Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
{{{\hat{y}}}_{i}} &= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
{{{\hat{y}}}_{5}} & = & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
& {{e}_{i}}= & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
& {{e}_{5}}= & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
& \hat{y}(47,31)= & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:14:47Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
\hat{y} &= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
& {{{\hat{y}}}_{i}}= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
& {{{\hat{y}}}_{5}}= & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
& {{e}_{i}}= & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
& {{e}_{5}}= & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
& \hat{y}(47,31)= & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:14:15Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
& \hat{y}= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
\hat{y} & = & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
& {{{\hat{y}}}_{i}}= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
& {{{\hat{y}}}_{5}}= & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
& {{e}_{i}}= & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
& {{e}_{5}}= & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
& \hat{y}(47,31)= & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Multiple Linear Regression Analysis

2012-08-24T06:13:30Z

Pengying niu: /* Estimating Regression Models Using Least Squares */

{{Template:Doebook|4}}

==Introduction==

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in DOE++ are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Chapter 9. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors.
ANOVA models are discussed in Chapter 6.

==Multiple Linear Regression Model==

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> .

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The model is linear because it is linear in the parameters <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> . The model describes a plane in the three dimensional space of <math>Y</math> , <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The parameter <math>{{\beta }_{0}}</math> is the intercept of this plane. Parameters <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> are referred to as partial regression coefficients. Parameter <math>{{\beta }_{1}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{1}}</math> when <math>{{x}_{2}}</math> is held constant. Parameter <math>{{\beta }_{2}}</math> represents the change in the mean response corresponding to a unit change in <math>{{x}_{2}}</math> when <math>{{x}_{1}}</math> is held constant.
Consider the following example of a multiple linear regression model with two predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> :

::<math>Y=30+5{{x}_{1}}+7{{x}_{2}}+\epsilon </math>

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is one. The regression plane corresponding to this model is shown in Figure TrueRegrPlane. Also shown is an observed data point and the corresponding random error, <math>\epsilon </math> . The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Section 5.MatrixApproach.

Figure ContourPlot1 shows the contour plot for the regression model of Eqn. (FirstOrderModelExample). The contour plot shows lines of constant mean response values as a function of <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.1.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

A cross-product term, <math>{{x}_{1}}{{x}_{2}}</math> , is included in the model. This term represents an interaction effect between the two variables <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation <math>Y=30+5{{x}_{1}}+7{{x}_{2}}+3{{x}_{1}}{{x}_{2}}+\epsilon </math> . The regression plane and contour plot for this model are shown in Figures RegrPlaneWInteraction and ContourPlotWInteraction, respectively.

[[Image:doe5.2.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+\epsilon</math>]]

Now consider the regression model shown next:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}x_{1}^{2}+{{\beta }_{3}}x_{1}^{3}+\epsilon </math>

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3x_{1}^{2}-5x_{2}^{2}+3{{x}_{1}}{{x}_{2}}+\epsilon </math>

[[Image:doe5.3.png|thumb|center|300px|Regression plane for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

[[Image:doe5.4.png|thumb|center|300px|Countour plot for the model <math>Y=30+5 x_1+7 x_2+3 x_1 x_2+\epsilon </math>]]

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in Figure PolynomialRegrSurface. Such regression models are used in RSM to find the optimum value of the response, <math>Y</math> (for details see Chapter 9). Notice that, although the shape of the regression surface is curvilinear, the regression model of Eqn. (SecondOrderModelEx) is still linear because the model is linear in the parameters. The contour plot for this model is shown in Figure ContourPlotPolynomialRegr.
All multiple linear regression models can be expressed in the following general form:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

where <math>k</math> denotes the number of terms in the model. For example, the model of Eqn. (SecondOrderModelEx) can be written in the general form using <math>{{x}_{3}}=x_{1}^{2}</math> , <math>{{x}_{4}}=x_{2}^{3}</math> and <math>{{x}_{5}}={{x}_{1}}{{x}_{2}}</math> as follows:

::<math>Y=500+5{{x}_{1}}+7{{x}_{2}}-3{{x}_{3}}-5{{x}_{4}}+3{{x}_{5}}+\epsilon </math>

==Estimating Regression Models Using Least Squares==

Consider a multiple linear regression model with <math>k</math> predictor variables:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+...+{{\beta }_{k}}{{x}_{k}}+\epsilon </math>

Let each of the <math>k</math> predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> , have <math>n</math> levels. Then <math>{{x}_{ij}}</math> represents the <math>i</math> th level of the <math>j</math> th predictor variable <math>{{x}_{j}}</math> . For example, <math>{{x}_{51}}</math> represents the fifth level of the first predictor variable <math>{{x}_{1}}</math> , while <math>{{x}_{19}}</math> represents the first level of the ninth predictor variable, <math>{{x}_{9}}</math> . Observations, <math>{{y}_{1}}</math> , <math>{{y}_{2}}</math> ... <math>{{y}_{n}}</math> , recorded for each of these <math>n</math> levels can be expressed in the following way:

::<math>\begin{align}
& {{y}_{1}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{11}}+{{\beta }_{2}}{{x}_{12}}+...+{{\beta }_{k}}{{x}_{1k}}+{{\epsilon }_{1}} \\
& {{y}_{2}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{21}}+{{\beta }_{2}}{{x}_{22}}+...+{{\beta }_{k}}{{x}_{2k}}+{{\epsilon }_{2}} \\
& & .. \\
& {{y}_{i}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{i1}}+{{\beta }_{2}}{{x}_{i2}}+...+{{\beta }_{k}}{{x}_{ik}}+{{\epsilon }_{i}} \\
& & .. \\
& {{y}_{n}}= & {{\beta }_{0}}+{{\beta }_{1}}{{x}_{n1}}+{{\beta }_{2}}{{x}_{n2}}+...+{{\beta }_{k}}{{x}_{nk}}+{{\epsilon }_{n}}
\end{align}</math>

[[Image:doe5.5.png|thumb|center|400px|Regression surface for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

[[Image:doe5.6.png|thumb|center|400px|Contour plot for the model <math>500+5 x_1+7 x_2-3 x_1^2-5 x_2^2+3 x_1 x_2+\epsilon </math>.]]

The system of <math>n</math> equations shown previously can be represented in matrix notation as follows:

::<math>y=X\beta +\epsilon </math>

:where

::<math>y=\left[ \begin{matrix}
{{y}_{1}} \\
{{y}_{2}} \\
. \\
. \\
. \\
{{y}_{n}} \\
\end{matrix} \right]\text{ }X=\left[ \begin{matrix}
1 & {{x}_{11}} & {{x}_{12}} & . & . & . & {{x}_{1n}} \\
1 & {{x}_{21}} & {{x}_{22}} & . & . & . & {{x}_{2n}} \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
. & . & . & {} & {} & {} & . \\
1 & {{x}_{n1}} & {{x}_{n2}} & . & . & . & {{x}_{nn}} \\
\end{matrix} \right]</math>

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{0}} \\
{{\beta }_{1}} \\
. \\
. \\
. \\
{{\beta }_{n}} \\
\end{matrix} \right]\text{ and }\epsilon =\left[ \begin{matrix}
{{\epsilon }_{1}} \\
{{\epsilon }_{2}} \\
. \\
. \\
. \\
{{\epsilon }_{n}} \\
\end{matrix} \right]</math>

The matrix <math>X</math> in Eqn. (TrueModelMatrixNotation) is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector <math>\beta </math> contains all the regression coefficients. To obtain the regression model, <math>\beta </math> should be known. <math>\beta </math> is estimated using least square estimates. The following equation is used:

::<math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math>

where <math>^{\prime }</math> represents the transpose of the matrix while <math>^{-1}</math> represents the matrix inverse. Knowing the estimates, <math>\hat{\beta }</math> , the multiple linear regression model can now be estimated as:

::<math>\hat{y}=X\hat{\beta }</math>

The estimated regression model is also referred to as the fitted model. The observations, <math>{{y}_{i}}</math> , may be different from the fitted values <math>{{\hat{y}}_{i}}</math> obtained from this model. The difference between these two values is the residual, <math>{{e}_{i}}</math> . The vector of residuals, <math>e</math> , is obtained as:

::<math>e=y-\hat{y}</math>

The fitted model of Eqn. (FittedValueMatrixNotation) can also be written as follows, using <math>\hat{\beta }={{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y</math> from Eqn. (LeastSquareEstimate):

::<math>\begin{align}
& \hat{y}= & X\hat{\beta } \\
& = & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & Hy
\end{align}</math>

where <math>H=X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}</math> . The matrix, <math>H</math> , is referred to as the hat matrix. It transforms the vector of the observed response values, <math>y</math> , to the vector of fitted values, <math>\hat{y}</math> .

'''Example 1'''

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> . Observations recorded for various levels of the two factors are shown in Table 5.1. The analyst wants to fit a first order regression model to the data. Interaction between <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

[[Image:doet5.1.png|thumb|center|400px|Observed yield data for various levels of two factors.]]

The data of Table 5.1 can be entered into DOE++ using the Multiple Regression tool as shown in Figure MLRTDataEntrySshot. A scatter plot for the data in Table 5.1 is shown in Figure ThreedScatterPlot. The first order regression model applicable to this data set having two predictor variables is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

where the dependent variable, <math>Y</math> , represents the yield and the predictor variables, <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> , represent the two factors respectively. The <math>X</math> and <math>y</math> matrices for the data can be obtained as:

::<math>X=\left[ \begin{matrix}
1 & 41.9 & 29.1 \\
1 & 43.4 & 29.3 \\
. & . & . \\
. & . & . \\
. & . & . \\
1 & 77.8 & 32.9 \\
\end{matrix} \right]\text{ }y=\left[ \begin{matrix}
251.3 \\
251.3 \\
. \\
. \\
. \\
349.0 \\
\end{matrix} \right]</math>

[[Image:doe5.7.png|thumb|center|400px|Multiple Regression tool in DOE++ with the data in Table 5.1.]]

[[Image:doe5.8.png|thumb|center|400px|Three dimensional scatter plot for the observed data in Table 5.1.]]

The least square estimates, <math>\hat{\beta }</math> , can now be obtained:

::<math>\begin{align}
\hat{\beta } &= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & {{\left[ \begin{matrix}
17 & 941 & 525.3 \\
941 & 54270 & 29286 \\
525.3 & 29286 & 16254 \\
\end{matrix} \right]}^{-1}}\left[ \begin{matrix}
4902.8 \\
276610 \\
152020 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]
\end{align}</math>

:Thus:

::<math>\hat{\beta }=\left[ \begin{matrix}
{{{\hat{\beta }}}_{0}} \\
{{{\hat{\beta }}}_{1}} \\
{{{\hat{\beta }}}_{2}} \\
\end{matrix} \right]=\left[ \begin{matrix}
-153.51 \\
1.24 \\
12.08 \\
\end{matrix} \right]</math>

and the estimated regression coefficients are <math>{{\hat{\beta }}_{0}}=-153.51</math> , <math>{{\hat{\beta }}_{1}}=1.24</math> and <math>{{\hat{\beta }}_{2}}=12.08</math> . The fitted regression model is:

::<math>\begin{align}
& \hat{y}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{1}}+{{{\hat{\beta }}}_{2}}{{x}_{2}} \\
& = & -153.5+1.24{{x}_{1}}+12.08{{x}_{2}}
\end{align}</math>

In DOE++, the fitted regression model can be viewed using the Show Analysis Summary icon in the Control Panel. The model is shown in Figure EquationScreenshot.

A plot of the fitted regression plane is shown in Figure FittedRegrModel. The fitted regression model can be used to obtain fitted values, <math>{{\hat{y}}_{i}}</math> , corresponding to an observed response value, <math>{{y}_{i}}</math> . For example, the fitted value corresponding to the fifth observation is:

[[Image:doe5.9.png|thumb|center|400px|Equation of the fitted regression model for the data in Table 5.1.]]

[[Image:doe5.10.png|thumb|center|400px|Fitted regression plane <math>\hat{y}=-153.5+1.24 x_1+12.08 x_2 </math> for the data of Table 5.1.]]

::<math>\begin{align}
& {{{\hat{y}}}_{i}}= & -153.5+1.24{{x}_{i1}}+12.08{{x}_{i2}} \\
& {{{\hat{y}}}_{5}}= & -153.5+1.24{{x}_{51}}+12.08{{x}_{52}} \\
& = & -153.5+1.24(47.3)+12.08(29.9) \\
& = & 266.3
\end{align}</math>

The observed fifth response value is <math>{{y}_{5}}=273.0</math> . The residual corresponding to this value is:

::<math>\begin{align}
& {{e}_{i}}= & {{y}_{i}}-{{{\hat{y}}}_{i}} \\
& {{e}_{5}}= & {{y}_{5}}-{{{\hat{y}}}_{5}} \\
& = & 273.0-266.3 \\
& = & 6.7
\end{align}</math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure DiagnosticSshot. The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of <math>{{x}_{1}}</math> and 31 units of <math>{{x}_{2}}</math> , the value is calculated using:

::<math>\begin{align}
& \hat{y}(47,31)= & -153.5+1.24(47)+12.08(31) \\
& = & 279.26
\end{align}</math>

===Properties of the Least Square Estimators, <math>\hat{\beta }</math>===
The least square estimates, <math>{{\hat{\beta }}_{0}}</math> , <math>{{\hat{\beta }}_{1}}</math> , <math>{{\hat{\beta }}_{2}}</math> ... <math>{{\hat{\beta }}_{k}}</math> , are unbiased estimators of <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> ... <math>{{\beta }_{k}}</math> , provided that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed. The variances of the <math>\hat{\beta }</math> s are obtained using the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

::<math>C={{\hat{\sigma }}^{2}}{{({{X}^{\prime }}X)}^{-1}}</math>

[[Image:doe5.11.png|thumb|center|400px|Fitted values and residuals for the data in Table 5.1.]]

<math>C</math> is a symmetric matrix whose diagonal elements, <math>{{C}_{jj}}</math> , represent the variance of the estimated <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> . The off-diagonal elements, <math>{{C}_{ij}}</math> , represent the covariance between the <math>i</math> th and <math>j</math> th estimated regression coefficients, <math>{{\hat{\beta }}_{i}}</math> and <math>{{\hat{\beta }}_{j}}</math> . The value of <math>{{\hat{\sigma }}^{2}}</math> is obtained using the error mean square, <math>M{{S}_{E}}</math> , which can be calculated as discussed in Section 5.MANOVA. The variance-covariance matrix for the data in Table 5.1 is shown in Figure VarCovMatrixSshot. It is available in DOE++ using the Show Analysis Summary icon in the Control Panel. Calculations to obtain the matrix are given in Example 3 in Section 5.tTest. The positive square root of <math>{{C}_{jj}}</math> represents the estimated standard deviation of the <math>j</math> th regression coefficient, <math>{{\hat{\beta }}_{j}}</math> , and is called the estimated standard error of <math>{{\hat{\beta }}_{j}}</math> (abbreviated <math>se({{\hat{\beta }}_{j}})</math> ).

::<math>se({{\hat{\beta }}_{j}})=\sqrt{{{C}_{jj}}}</math>

[[Image:doe5.12.png|thumb|center|400px|The variance-covariance matrix for the data in Table 5.1.]]

==Hypothesis Tests in Multiple Linear Regression==

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, <math>{{\epsilon }_{i}}</math> , are normally and independently distributed with a mean of zero and variance of <math>{{\sigma }^{2}}</math> .
Three types of hypothesis tests can be carried out for multiple linear regression models:
:• Test for significance of regression

This test checks the significance of the whole regression model.

:• <math>t</math> test

This test checks the significance of individual regression coefficients.

:• Partial <math>F</math> test

This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

===Test for Significance of Regression===

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{1}}={{\beta }_{2}}=...={{\beta }_{k}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0\text{ for at least one }j
\end{align}</math>

The test for <math>{{H}_{0}}</math> is carried out using the following statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

where <math>M{{S}_{R}}</math> is the regression mean square and <math>M{{S}_{E}}</math> is the error mean square. If the null hypothesis, <math>{{H}_{0}}</math> , is true then the statistic <math>{{F}_{0}}</math> follows the <math>F</math> distribution with <math>k</math> degrees of freedom in the numerator and <math>n-</math> ( <math>k+1</math> ) degrees of freedom in the denominator. The null hypothesis, <math>{{H}_{0}}</math> , is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,k,n-(k+1)}}</math>

====Calculation of the Statistic <math>{{F}_{0}}</math>====

To calculate the statistic <math>{{F}_{0}}</math> , the mean squares <math>M{{S}_{R}}</math> and <math>M{{S}_{E}}</math> must be known. As explained in Chapter 4, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, <math>M{{S}_{T}}</math> , is obtained as follows:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}</math>

where <math>S{{S}_{T}}</math> is the total sum of squares and <math>dof(S{{S}_{T}})</math> is the number of degrees of freedom associated with <math>S{{S}_{T}}</math> . In multiple linear regression, the following equation is used to calculate <math>S{{S}_{T}}</math> :

::<math>S{{S}_{T}}={{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations (that was defined in Section 5.MatrixApproach), <math>I</math> is the identity matrix of order <math>n</math> and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is ( <math>n-1</math> ). Knowing <math>S{{S}_{T}}</math> and <math>dof(S{{S}_{T}})</math> the total mean square, <math>M{{S}_{T}}</math> , can be calculated.

The regression mean square, <math>M{{S}_{R}}</math> , is obtained by dividing the regression sum of squares, <math>S{{S}_{R}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{R}})</math> , as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}</math>

The regression sum of squares, <math>S{{S}_{R}}</math> , is calculated using the following equation:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

where <math>n</math> is the total number of observations, <math>y</math> is the vector of observations, <math>H</math> is the hat matrix (that was defined in Section 5.MatrixApproach) and <math>J</math> represents an <math>n\times n</math> square matrix of ones. The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>k</math> , where <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{R}}</math> and <math>dof(S{{S}_{R}})</math> the regression mean square, <math>M{{S}_{R}}</math> , can be calculated.
The error mean square, <math>M{{S}_{E}}</math> , is obtained by dividing the error sum of squares, <math>S{{S}_{E}}</math> , by the respective degrees of freedom, <math>dof(S{{S}_{E}})</math> , as follows:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}</math>

The error sum of squares, <math>S{{S}_{E}}</math> , is calculated using the following equation:

::<math>S{{S}_{E}}={{y}^{\prime }}(I-H)y</math>

where <math>y</math> is the vector of observations, <math>I</math> is the identity matrix of order <math>n</math> and <math>H</math> is the hat matrix. The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>n-(k+1)</math> , where <math>n</math> is the total number of observations and <math>k</math> is the number of predictor variables in the model. Knowing <math>S{{S}_{E}}</math> and <math>dof(S{{S}_{E}})</math> , the error mean square, <math>M{{S}_{E}}</math> , can be calculated. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error terms, <math>{{\epsilon }_{i}}</math> .

::<math>{{\hat{\sigma }}^{2}}=M{{S}_{E}}</math>

'''Example 2'''

The test for the significance of regression, for the regression model obtained for the data in Table 5.1, is illustrated in this example. The null hypothesis for the model is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}={{\beta }_{2}}=0</math>

The statistic to test <math>{{H}_{0}}</math> is:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}</math>

To calculate <math>{{F}_{0}}</math> , first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic <math>{{F}_{0}}</math> to carry out the significance test.
The regression sum of squares, <math>S{{S}_{R}}</math> , can be obtained as:

::<math>S{{S}_{R}}={{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y</math>

The hat matrix, <math>H</math> is calculated as follows using the design matrix <math>X</math> from Example 1:

::<math>\begin{align}
& H= & X{{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }} \\
& = & \left[ \begin{matrix}
0.27552 & 0.25154 & . & . & -0.04030 \\
0.25154 & 0.23021 & . & . & -0.029120 \\
. & . & . & . & . \\
. & . & . & . & . \\
-0.04030 & -0.02920 & . & . & 0.30115 \\
\end{matrix} \right]
\end{align}</math>

Knowing <math>y</math> , <math>H</math> and <math>J</math> , the regression sum of squares, <math>S{{S}_{R}}</math> , can be calculated:

::<math>\begin{align}
& S{{S}_{R}}= & {{y}^{\prime }}\left[ H-(\frac{1}{n})J \right]y \\
& = & 12816.35
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{R}}</math> is <math>k</math> , which equals to a value of two since there are two predictor variables in the data in Table 5.1. Therefore, the regression mean square is:

::<math>\begin{align}
& M{{S}_{R}}= & \frac{S{{S}_{R}}}{dof(S{{S}_{R}})} \\
& = & \frac{12816.35}{2} \\
& = & 6408.17
\end{align}</math>

Similarly to calculate the error mean square, <math>M{{S}_{E}}</math> , the error sum of squares, <math>S{{S}_{E}}</math> , can be obtained as:

::<math>\begin{align}
& S{{S}_{E}}= & {{y}^{\prime }}\left[ I-H \right]y \\
& = & 423.37
\end{align}</math>

The degrees of freedom associated with <math>S{{S}_{E}}</math> is <math>n-(k+1)</math> . Therefore, the error mean square, <math>M{{S}_{E}}</math> , is:

::<math>\begin{align}
& M{{S}_{E}}= & \frac{S{{S}_{E}}}{dof(S{{S}_{E}})} \\
& = & \frac{S{{S}_{E}}}{(n-(k+1))} \\
& = & \frac{423.37}{(17-(2+1))} \\
& = & 30.24
\end{align}</math>

The statistic to test the significance of regression can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{6408.17}{423.37/(17-3)} \\
& = & 211.9
\end{align}</math>

The critical value for this test, corresponding to a significance level of 0.1, is:

::<math>\begin{align}
& {{f}_{\alpha ,k,n-(k+1)}}= & {{f}_{0.1,2,14}} \\
& = & 2.726
\end{align}</math>

Since <math>{{f}_{0}}>{{f}_{0.1,2,14}}</math> , <math>{{H}_{0}}\ \ :</math> <math>{{\beta }_{1}}={{\beta }_{2}}=0</math> is rejected and it is concluded that at least one coefficient out of <math>{{\beta }_{1}}</math> and <math>{{\beta }_{2}}</math> is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in Table 5.1. The analysis of variance is summarized in Table 5.2.

[[Image:doet5.2.png|thumb|center|300px|ANOVA table for the significance of regression test in Example 2.]]

===Test on Individual Regression Coefficients ( <math>t</math> Test)===
The <math>t</math> test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, <math>{{\beta }_{j}}</math> , are:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{j}}=0 \\
& {{H}_{1}}: & {{\beta }_{j}}\ne 0
\end{align}</math>

The test statistic for this test is based on the <math>t</math> distribution (and is similar to the one used in the case of simple linear regression models in Chapter 4):

::<math>{{T}_{0}}=\frac{{{{\hat{\beta }}}_{j}}}{se({{{\hat{\beta }}}_{j}})}</math>

where the standard error, <math>se({{\hat{\beta }}_{j}})</math> , is obtained from Eqn. (StandardErrorBetaJ). The analyst would fail to reject the null hypothesis if the test statistic, calculated using Eqn. (TtestStatistic), lies in the acceptance region:

::<math>-{{t}_{\alpha /2,n-2}}<{{T}_{0}}<{{t}_{\alpha /2,n-2}}</math>

This test measures the contribution of a variable while the remaining variables are included in the model. For the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{1}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> , if the test is carried out for <math>{{\beta }_{1}}</math> , then the test will check the significance of including the variable <math>{{x}_{1}}</math> in the model that contains <math>{{x}_{2}}</math> and <math>{{x}_{3}}</math> (i.e. the model <math>\hat{y}={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{2}}{{x}_{2}}+{{\hat{\beta }}_{3}}{{x}_{3}}</math> ). Hence the test is also referred to as partial or marginal test. In DOE++, this test is displayed in the Regression Information table.

'''Example 3'''

The test to check the significance of the estimated regression coefficients for the data in Table 5.1 is illustrated in this example. The null hypothesis to test the coefficient <math>{{\beta }_{2}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math>

The null hypothesis to test <math>{{\beta }_{1}}</math> can be obtained in a similar manner. To calculate the test statistic, <math>{{T}_{0}}</math> , we need to calculate the standard error using Eqn. (StandardErrorBetaJ).
In Example 2, the value of the error mean square, <math>M{{S}_{E}}</math> , was obtained as 30.24. The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> .

:Therefore:

::<math>\begin{align}
& {{{\hat{\sigma }}}^{2}}= & M{{S}_{E}} \\
& = & 30.24
\end{align}</math>

The variance-covariance matrix of the estimated regression coefficients is:

::<math>\begin{align}
& C= & {{{\hat{\sigma }}}^{2}}{{({{X}^{\prime }}X)}^{-1}} \\
& = & 30.24\left[ \begin{matrix}
336.5 & 1.2 & -13.1 \\
1.2 & 0.005 & -0.049 \\
-13.1 & -0.049 & 0.5 \\
\end{matrix} \right] \\
& = & \left[ \begin{matrix}
10176.75 & 37.145 & -395.83 \\
37.145 & 0.1557 & -1.481 \\
-395.83 & -1.481 & 15.463 \\
\end{matrix} \right]
\end{align}</math>

From the diagonal elements of <math>C</math> , the estimated standard error for <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{2}}</math> is:

::<math>\begin{align}
& se({{{\hat{\beta }}}_{1}})= & \sqrt{0.1557}=0.3946 \\
& se({{{\hat{\beta }}}_{2}})= & \sqrt{15.463}=3.93
\end{align}</math>

The corresponding test statistics for these coefficients are:

::<math>\begin{align}
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{1}}}}= & \frac{{{{\hat{\beta }}}_{1}}}{se({{{\hat{\beta }}}_{1}})}=\frac{1.24}{0.3946}=3.1393 \\
& {{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}= & \frac{{{{\hat{\beta }}}_{2}}}{se({{{\hat{\beta }}}_{2}})}=\frac{12.08}{3.93}=3.0726
\end{align}</math>

The critical values for the present <math>t</math> test at a significance of 0.1 are:

::<math>\begin{align}
& {{t}_{\alpha /2,n-(k+1)}}= & {{t}_{0.05,14}}=1.761 \\
& -{{t}_{\alpha /2,n-(k+1)}}= & -{{t}_{0.05,14}}=-1.761
\end{align}</math>

Considering <math>{{\hat{\beta }}_{2}}</math> , it can be seen that <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}</math> does not lie in the acceptance region of <math>-{{t}_{0.05,14}}<{{t}_{0}}<{{t}_{0.05,14}}</math> . The null hypothesis, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{2}}=0</math> , is rejected and it is concluded that <math>{{\beta }_{2}}</math> is significant at <math>\alpha =0.1</math> . This conclusion can also be arrived at using the <math>p</math> value noting that the hypothesis is two-sided. The <math>p</math> value corresponding to the test statistic, <math>{{({{t}_{0}})}_{{{{\hat{\beta }}}_{2}}}}=</math> <math>3.0726</math> , based on the <math>t</math> distribution with 14 degrees of freedom is:

::<math>\begin{align}
& p\text{ }value= & 2\times (1-P(T\le |{{t}_{0}}|) \\
& = & 2\times (1-0.9959) \\
& = & 0.0083
\end{align}</math>

Since the <math>p</math> value is less than the significance, <math>\alpha =0.1</math> , it is concluded that <math>{{\beta }_{2}}</math> is significant. The hypothesis test on <math>{{\beta }_{1}}</math> can be carried out in a similar manner.

As explained in Chapter 4, in DOE++, the information related to the <math>t</math> test is displayed in the Regression Information table as shown in Figure RegrInfoSshot. In this table, the <math>t</math> test for <math>{{\beta }_{2}}</math> is displayed in the row for the term Factor 2 because <math>{{\beta }_{2}}</math> is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the <math>t</math> test and the <math>p</math> value for the <math>t</math> test, respectively. These values have been calculated for <math>{{\beta }_{2}}</math> in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated using Eqn. (LeastSquareEstimate) as shown in Example

:1. The Effect column represents values obtained by multiplying the coefficients by a factor of
:2. This value is useful in the case of two factor experiments and is explained in Chapter 7.

Columns labeled Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Section 5.RegrCoeffCI. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in
Section 5.MultiCollinearity.

[[Image:doe5.13.png|thumb|center|400px|Regression results for the data in Table 5.1.]]
 

===Test on Subsets of Regression Coefficients (Partial <math>F</math> Test)===

This test can be considered to be the general form of the <math>t</math> test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, <math>S{{S}_{R}}</math> . The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares.
Assume that the vector of the regression coefficients, <math>\beta </math> , for the multiple linear regression model, <math>y=X\beta +\epsilon </math> , is partitioned into two vectors with the second vector, <math>{{\beta }_{2}}</math> , containing the last <math>r</math> regression coefficients, and the first vector, <math>{{\beta }_{1}}</math> , containing the first ( <math>k+1-r</math> ) coefficients as follows:

::<math>\beta =\left[ \begin{matrix}
{{\beta }_{1}} \\
{{\beta }_{2}} \\
\end{matrix} \right]</math>

:with:

::<math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}}...{{\beta }_{k-r}}{]}'\text{ and }{{\beta }_{2}}=[{{\beta }_{k-r+1}},{{\beta }_{k-r+2}}...{{\beta }_{k}}{]}'\text{ }</math>

The hypothesis statements to test the significance of adding the regression coefficients in <math>{{\beta }_{2}}</math> to a model containing the regression coefficients in <math>{{\beta }_{1}}</math> may be written as:

::<math>\begin{align}
& {{H}_{0}}: & {{\beta }_{2}}=0 \\
& {{H}_{1}}: & {{\beta }_{2}}\ne 0
\end{align}</math>

The test statistic for this test follows the <math>F</math> distribution and can be calculated as follows:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> is the the increase in the regression sum of squares when the variables corresponding to the coefficients in <math>{{\beta }_{2}}</math> are added to a model already containing <math>{{\beta }_{1}}</math> , and <math>M{{S}_{E}}</math> is obtained from Eqn. (ErrorMeanSquare). The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, <math>{{H}_{0}}</math> , is rejected if <math>{{F}_{0}}>{{f}_{\alpha ,r,n-(k+1)}}</math> . Rejection of <math>{{H}_{0}}</math> leads to the conclusion that at least one of the variables in <math>{{x}_{k-r+1}}</math> , <math>{{x}_{k-r+2}}</math> ... <math>{{x}_{k}}</math> contributes significantly to the regression model. In DOE++, the results from the partial <math>F</math> test are displayed in the ANOVA table.

===Types of Extra Sum of Squares===
The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic of Eqn. (PartialFtest). In DOE++, selection for the type of extra sum of squares is available in the Options tab of the Control Panel as shown in Figure SSselectionSshot. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

[[Image:doe5.14.png|thumb|center|500px|Selection of the type of extra sum of squared in DOE++.]]

====Partial Sum of Squares====
The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

Assume that we need to know the partial sum of squares for <math>{{\beta }_{2}}</math> . The partial sum of squares for <math>{{\beta }_{2}}</math> is the increase in the regression sum of squares when <math>{{\beta }_{2}}</math> is added to the model. This increase is the difference in the regression sum of squares for the full model of Eqn. (PartialSSFullModel) and the model that includes all terms except <math>{{\beta }_{2}}</math> . These terms are <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> and <math>{{\beta }_{12}}</math> . The model that contains these terms is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+\epsilon </math>

The partial sum of squares for <math>{{\beta }_{2}}</math> can be represented as <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})</math> and is calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})= & S{{S}_{R}}\text{ for Eqn}\text{. () }-S{{S}_{R}}\text{ for Eqn}\text{. ()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{2}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{12}}{]}'</math> . It can be noted that for the partial sum of squares <math>{{\beta }_{1}}</math> contains all coefficients other than the coefficient being tested.

DOE++ has the partial sum of squares as the default selection. This is because the <math>t</math> test explained in Section 5.tTest is a partial test, i.e. the <math>t</math> test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the <math>t</math> test are displayed in the Regression Information table. The results from the partial <math>F</math> test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table.
The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

'''Example 4'''

This example illustrates the partial <math>F</math> test using the partial sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the partial sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The partial sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the full model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , and the regression sum of squares for the model excluding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> . The regression sum of squares for the full model can be obtained using Eqn. (TotalSumofSquares) and has been calculated in Example 2 as <math>12816.35</math> . Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})=12816.35</math>

The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> , is obtained by dropping the second column in the design matrix of the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The second column of <math>X</math> corresponds to the coefficient <math>{{\beta }_{1}}</math> which is no longer in the model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{2}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}={{X}_{{{\beta }_{0}},{{\beta }_{2}}}}{{(X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{2}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{2}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{2}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math> , can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{2}}}}-(\frac{1}{n})J \right]y \\
& = & 12518.32
\end{align}</math>

Therefore, the partial sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}})-S{{S}_{R}}({{\beta }_{0}},{{\beta }_{2}}) \\
& = & 12816.35-12518.32 \\
& = & 298.03
\end{align}</math>

Knowing the partial sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{298.03/1}{30.24} \\
& = & 9.855
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.9928 \\
& = & 0.0072
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. In the results obtained from DOE++, the calculations for this test are displayed in the ANOVA table as shown in Figure AnovaTableSshot. Note that the conclusion obtained in this example can also be obtained using the <math>t</math> test as explained in Example 3 in Section 5.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the variables included in the multiple linear regression model.

====Sequential Sum of Squares====
The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+{{\beta }_{23}}{{x}_{2}}{{x}_{3}}+{{\beta }_{123}}{{x}_{1}}{{x}_{2}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> is the increase in the sum of squares when <math>{{\beta }_{13}}</math> is added to the model observing the sequence of Eqn. (SeqSSEqn). Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after <math>{{\beta }_{13}}</math> was added and the regression sum of squares for the model before <math>{{\beta }_{13}}</math> was added to the model. The model after <math>{{\beta }_{13}}</math> is added is as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+{{\beta }_{13}}{{x}_{1}}{{x}_{3}}+\epsilon </math>

[[Image:doe5.15.png|thumb|center|500px|ANOVA results for the data in Table 5.1.]]

This is because to maintain the sequence of Eqn. (SeqSSEqn) all coefficients preceding <math>{{\beta }_{13}}</math> must be included in the model. These are the coefficients <math>{{\beta }_{0}}</math> , <math>{{\beta }_{1}}</math> , <math>{{\beta }_{2}}</math> , <math>{{\beta }_{12}}</math> and <math>{{\beta }_{3}}</math> .
Similarly the model before <math>{{\beta }_{13}}</math> is added must contain all coefficients of Eqn. (SeqSSEqnafter) except <math>{{\beta }_{13}}</math> . This model can be obtained as follows:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+{{\beta }_{12}}{{x}_{1}}{{x}_{2}}+{{\beta }_{3}}{{x}_{3}}+\epsilon </math>

The sequential sum of squares for <math>{{\beta }_{13}}</math> can be calculated as follows:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{13}}|{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})= & S{{S}_{R}}\text{ for Eqn}\text{.()}-S{{S}_{R}}\text{ for Eqn}\text{.()} \\
& = & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}},{{\beta }_{13}})- \\
& & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}})
\end{align}</math>

For the present case, <math>{{\beta }_{2}}=[{{\beta }_{13}}{]}'</math> and <math>{{\beta }_{1}}=[{{\beta }_{0}},{{\beta }_{1}},{{\beta }_{2}},{{\beta }_{12}},{{\beta }_{3}}{]}'</math> . It can be noted that for the sequential sum of squares <math>{{\beta }_{1}}</math> contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

'''Example 5'''

This example illustrates the partial <math>F</math> test using the sequential sum of squares. The test is conducted for the coefficient <math>{{\beta }_{1}}</math> corresponding to the predictor variable <math>{{x}_{1}}</math> for the data in Table 5.1. The regression model used for this data set in Example 1 is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The null hypothesis to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math>

The statistic to test this hypothesis is:

::<math>{{F}_{0}}=\frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}}</math>

where <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> represents the sequential sum of squares for <math>{{\beta }_{1}}</math> , <math>r</math> represents the number of degrees of freedom for <math>S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})</math> (which is one because there is just one coefficient, <math>{{\beta }_{1}}</math> , being tested) and <math>M{{S}_{E}}</math> is the error mean square that can obtained using Eqn. (ErrorMeanSquare) and has been calculated in Example 2 as 30.24.

The sequential sum of squares for <math>{{\beta }_{1}}</math> is the difference between the regression sum of squares for the model after adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , and the regression sum of squares for the model before adding <math>{{\beta }_{1}}</math> , <math>Y={{\beta }_{0}}+\epsilon </math> .
The regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> is obtained as shown next. First the design matrix for this model, <math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> , is obtained by dropping the third column in the design matrix for the full model, <math>X</math> (the full design matrix, <math>X</math> , was obtained in Example 1). The third column of <math>X</math> corresponds to coefficient <math>{{\beta }_{2}}</math> which is no longer used in the present model. Therefore, the design matrix for the model, <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> , is:

::<math>{{X}_{{{\beta }_{0}},{{\beta }_{1}}}}=\left[ \begin{matrix}
1 & 41.9 \\
1 & 43.4 \\
. & . \\
. & . \\
1 & 77.8 \\
\end{matrix} \right]</math>

The hat matrix corresponding to this design matrix is <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> . It can be calculated using <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}={{X}_{{{\beta }_{0}},{{\beta }_{1}}}}{{(X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }{{X}_{{{\beta }_{0}},{{\beta }_{1}}}})}^{-1}}X_{{{\beta }_{0}},{{\beta }_{1}}}^{\prime }</math> . Once <math>{{H}_{{{\beta }_{0}},{{\beta }_{1}}}}</math> is known, the regression sum of squares for the model <math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+\epsilon </math> can be calculated using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})= & {{y}^{\prime }}\left[ {{H}_{{{\beta }_{0}},{{\beta }_{1}}}}-(\frac{1}{n})J \right]y \\
& = & 12530.85
\end{align}</math>

[[Image:doe5.16.png|thumb|center|500px|Sequential sum of squares for the data in Table 5.1.]]

The regression sum of squares for the model <math>Y={{\beta }_{0}}+\epsilon </math> is equal to zero since this model does not contain any variables. Therefore:

::<math>S{{S}_{R}}({{\beta }_{0}})=0</math>

The sequential sum of squares for <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})= & S{{S}_{R}}({{\beta }_{0}},{{\beta }_{1}})-S{{S}_{R}}({{\beta }_{0}}) \\
& = & 12530.85-0 \\
& = & 12530.85
\end{align}</math>

Knowing the sequential sum of squares, the statistic to test the significance of <math>{{\beta }_{1}}</math> is:

::<math>\begin{align}
& {{f}_{0}}= & \frac{S{{S}_{R}}({{\beta }_{2}}|{{\beta }_{1}})/r}{M{{S}_{E}}} \\
& = & \frac{12530.85/1}{30.24} \\
& = & 414.366
\end{align}</math>

The <math>p</math> value corresponding to this statistic based on the <math>F</math> distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 8.46\times {{10}^{-12}}
\end{align}</math>

Assuming that the desired significance is 0.1, since <math>p</math> value < 0.1, <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it can be concluded that <math>{{\beta }_{1}}</math> is significant. The test for <math>{{\beta }_{2}}</math> can be carried out in a similar manner. This result is shown in Figure SequentialSshot.

==Confidence Intervals in Multiple Linear Regression==

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Chapter 4.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on the regression coefficient, <math>{{\beta }_{j}}</math> , is obtained as follows:

::<math>{{\hat{\beta }}_{j}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{C}_{jj}}}</math>

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low CI and High CI columns as shown in Figure RegrInfoSshot.
Confidence Interval on Fitted Values, <math>{{\hat{y}}_{i}}</math>
A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is given by:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}x_{i}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{i}}}</math>

:where:

::<math>{{x}_{i}}=\left[ \begin{matrix}
1 \\
{{x}_{i1}} \\
. \\
. \\
. \\
{{x}_{ik}} \\
\end{matrix} \right]</math>

In Example 1 (Section 5.MatrixApproach), the fitted value corresponding to the fifth observation was calculated as <math>{{\hat{y}}_{5}}=266.3</math> . The 90% confidence interval on this value can be obtained as shown in Figure CIfittedvalueSshot. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation in Table 5.1.

[[Image:doe5.17.png|thumb|center|500px|Confidence interval for the fitted value corresponding to the fifth observation in Table 5.1.]]

===Confidence Interval on New Observations===

As explained in Chapter 4, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100( <math>1-\alpha </math> ) percent confidence interval on a new observation, <math>{{\hat{y}}_{p}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-(k+1)}}\sqrt{{{{\hat{\sigma }}}^{2}}(1+x_{p}^{\prime }{{({{X}^{\prime }}X)}^{-1}}{{x}_{p}})}</math>

where:

::<math>{{x}_{p}}=\left[ \begin{matrix}
1 \\
{{x}_{p1}} \\
. \\
. \\
. \\
{{x}_{pk}} \\
\end{matrix} \right]</math>

<math>{{x}_{p1}}</math> ,..., <math>{{x}_{pk}}</math> are the levels of the predictor variables at which the new observation, <math>{{\hat{y}}_{p}}</math> , needs to be obtained.

[[Image:doe5.18.png|thumb|center|400px|Predicted values and region of model application in multiple linear regression.]]

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider Figure JointRegion where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to <math>p</math> th level of first predictor variable, <math>{{x}_{1}}</math> , and <math>p</math> th level of the second predictor variable, <math>{{x}_{2}}</math> , does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

==Measures of Model Adequacy==

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

===Coefficient of Multiple Determination, <math>{{R}^{2}}</math>===

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}}{S{{S}_{T}}}
\end{align}</math>

<math>{{R}^{2}}</math> indicates the amount of total variability explained by the regression model. The positive square root of <math>{{R}^{2}}</math> is called the multiple correlation coefficient and measures the linear association between <math>Y</math> and the predictor variables, <math>{{x}_{1}}</math> , <math>{{x}_{2}}</math> ... <math>{{x}_{k}}</math> .

The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted <math>{{R}^{2}}</math> statistic defined as follows:

::<math>\begin{align}
& R_{adj}^{2}= & 1-\frac{M{{S}_{E}}}{M{{S}_{T}}} \\
& = & 1-\frac{S{{S}_{E}}/(n-(k+1))}{S{{S}_{T}}/(n-1)} \\
& = & 1-(\frac{n-1}{n-(k+1)})(1-{{R}^{2}})
\end{align}</math>

The adjusted <math>{{R}^{2}}</math> only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of <math>R_{adj}^{2}</math> .

In DOE++, <math>{{R}^{2}}</math> and <math>R_{adj}^{2}</math> values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Chapter 4, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, <math>{{e}_{i}}</math> , in Eqn. (ErrorSumofSquares). The PRESS residual, <math>{{e}_{(i)}}</math> , for a particular observation, <math>{{y}_{i}}</math> , is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, <math>{{\hat{y}}_{p}}</math> , corresponding to the observation in question, <math>{{y}_{i}}</math> , is obtained based on the new regression model. The difference between <math>{{y}_{i}}</math> and <math>{{\hat{y}}_{p}}</math> gives <math>{{e}_{(i)}}</math> . The PRESS residual, <math>{{e}_{(i)}}</math> , can also be obtained using <math>{{h}_{ii}}</math> , the diagonal element of the hat matrix, <math>H</math> , as follows:

::<math>{{e}_{(i)}}=\frac{{{e}_{i}}}{1-{{h}_{ii}}}</math>

R-sq(pred), also referred to as prediction <math>{{R}^{2}}</math> , is obtained using PRESS as shown next:

::<math>R_{pred}^{2}=1-\frac{PRESS}{S{{S}_{T}}}</math>

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. Figure RSqadjSshot. shows these values for the data in Table 5.1. The values indicate that the regression model fits the data well and also predicts well.

===Residual Analysis===

Plots of residuals, <math>{{e}_{i}}</math> , similar to the ones discussed in the previous chapter for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, <math>{{d}_{i}}</math> , are obtained using the following equation:

::<math>\begin{align}
& {{d}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}}}
\end{align}</math>

[[Image:doe5.19.png|thumb|center|400px|Coefficient of multiple determination and related results for the data in Table 5.1.]]

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, <math>{{r}_{i}}</math> , are used in their place. Studentized residuals are calculated as follows:

::<math>\begin{align}
& {{r}_{i}}= & \frac{{{e}_{i}}}{\sqrt{{{{\hat{\sigma }}}^{2}}(1-{{h}_{ii}})}} \\
& = & \frac{{{e}_{i}}}{\sqrt{M{{S}_{E}}(1-{{h}_{ii}})}}
\end{align}</math>

where <math>{{h}_{ii}}</math> is the <math>i</math> th diagonal element of the hat matrix, <math>H</math> . External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Section 5.Rsquare. The reason for using the external studentized residuals is that if the <math>i</math> th observation is an outlier, it may influence the fitted model. In this case, the residual <math>{{e}_{i}}</math> will be small and may not disclose that <math>i</math> th observation is an outlier. The external studentized residual for the <math>i</math> th observation, <math>{{t}_{i}}</math> , is obtained as follows:

::<math>{{t}_{i}}={{e}_{i}}{{\left[ \frac{n-k}{S{{S}_{E}}(1-{{h}_{ii}})-e_{i}^{2}} \right]}^{0.5}}</math>

Residual values for the data of Table 5.1 are shown in Figure ResidualSshot. These values are available using the Diagnostics icon in the Control Panel. Standardized residual plots for the data are shown in Figures Res1NPP to ResVsRuns. DOE++ compares the residual values to the critical values on the <math>t</math> distribution for studentized and external studentized residuals. For other residuals the normal distribution is used. For example, for the data in Table 5.1, the critical values on the <math>t</math> distribution at a significance of 0.1 are <math>{{t}_{0.05,14}}=1.761</math> and <math>-{{t}_{0.05,14}}=-1.761</math> (as calculated in Example 3, Section 5.tTest). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in Figures ResVsFitted and ResVsRuns.

===Outlying <math>x</math> Observations===

Residuals help to identify outlying <math>y</math> observations. Outlying <math>x</math> observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, <math>{{h}_{ii}}</math> . The <math>{{h}_{ii}}</math> values always lie between 0 and 1. Values of <math>{{h}_{ii}}</math> greater than <math>2(k+1)/n</math> are considered to be indicators of outlying <math>x</math> observations.

===Influential Observations Detection===

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

::<math>{{D}_{i}}=\frac{r_{i}^{2}}{(k+1)}\left[ \frac{{{h}_{ii}}}{(1-{{h}_{ii}})} \right]</math>

To use Cook's distance measure, the <math>{{D}_{i}}</math> values are compared to percentile values on the <math>F</math> distribution with <math>(k+1,n-(k+1))</math> degrees of freedom. If the percentile value is less than 10 or 20 percent, then the <math>i</math> th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the <math>i</math> th case is influential, and fitted values with and without the <math>i</math> th case will differ substantially.[Kutner]

'''Example 6'''

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data in Table 5.1. The remaining values along with the leverage values are shown in Figure CookSshot.
The standardized residual corresponding to the first observation is:

[[Image:doe5.20.png|thumb|center|400px|Residual values for the data in Table 5.1.]]

[[Image:doe5.21.png|thumb|center|400px|Residual probability plot for the data in Table 5.1.]]

[[Image:doe5.22.png|thumb|center|400px|Residual versus fitted values plot for the data in Table 5.1.]]

[[Image:doe5.23.png|thumb|center|400px|Residual versus run order plot for the data in Table 5.1.]]

::<math>\begin{align}
& {{r}_{1}}= & \frac{{{e}_{1}}}{\sqrt{M{{S}_{E}}(1-{{h}_{11}})}} \\
& = & \frac{1.3127}{\sqrt{30.3(1-0.2755)}} \\
& = & 0.2804
\end{align}</math>

Cook's distance measure for the first observation can now be calculated as:

::<math>\begin{align}
& {{D}_{1}}= & \frac{r_{1}^{2}}{(k+1)}\left[ \frac{{{h}_{11}}}{(1-{{h}_{11}})} \right] \\
& = & \frac{{{0.2804}^{2}}}{(2+1)}\left[ \frac{0.2755}{(1-0.2755)} \right] \\
& = & 0.01
\end{align}</math>

The 50th percentile value for <math>{{F}_{3,14}}</math> is 0.83. Since all <math>{{D}_{i}}</math> values are less than this value there are no influential observations.

[[Image:doe5.24.png|thumb|center|400px|Leverage and Cook's distance measure for the data in Table 5.1.]]

===Lack-of-Fit Test===

The lack-of-fit test for simple linear regression discussed in Chapter 4 may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for <math>m</math> replicates may be collected as follows for all <math>n</math> levels of the predictor variables:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1m}}\text{ }m\text{ repeated observations at the first level } \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2m}}\text{ }m\text{ repeated observations at the second level} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{im}}\text{ }m\text{ repeated observations at the }i\text{th level} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{nm}}\text{ }m\text{ repeated observations at the }n\text{th level }
\end{align}</math>

The sum of squares due to pure error, <math>S{{S}_{PE}}</math> , can be obtained as discussed in the previous chapter as:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{m}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Knowing <math>S{{S}_{PE}}</math> , sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> , can be obtained as:

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> are:

<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & n-(k+1)-(nm-n)
\end{align}</math>

The test statistic for the lack-of-fit test is:

::<math>\begin{align}
& {{F}_{0}}= & \frac{S{{S}_{LOF}}/dof(S{{S}_{LOF}})}{S{{S}_{PE}}/dof(S{{S}_{PE}})} \\
& = & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}}
\end{align}</math>

==Other Topics in Multiple Linear Regression==

===Polynomial Regression Models===

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{11}}x_{1}^{2}+\epsilon </math>

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

::<math>coded\text{ }value=\frac{actual\text{ }value-mean}{half\text{ }of\text{ }range}</math>

The reason for using coded predictor variables is that many times <math>x</math> and <math>{{x}^{2}}</math> are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the <math>{{({{X}^{\prime }}X)}^{-1}}</math> matrix to obtain the estimates, <math>\hat{\beta }</math> , of the regression coefficients using Eqn. (LeastSquareEstimate).

===Qualitative Factors===

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

::<math>{{x}_{1}}=\{\begin{array}{*{35}{l}}
1\text{ Female} \\
0\text{ Male} \\
\end{array}</math>

In general ( <math>n-1</math> ) indicator variables are required to represent a qualitative factor with <math>n</math> levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=0\text{ Machine Type III}
\end{align}</math>

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

::<math>\begin{align}
& {{x}_{1}}= & 1,\text{ }{{x}_{2}}=0\text{ Machine Type I} \\
& {{x}_{1}}= & 0,\text{ }{{x}_{2}}=1\text{ Machine Type II} \\
& {{x}_{1}}= & -1,\text{ }{{x}_{2}}=-1\text{ Machine Type III}
\end{align}</math>

Indicator variables are also referred to as dummy variables or binary variables.

'''Example 7'''

Consider data from two types of reactors of a chemical process shown in Table 5.3 where the yield values are recorded for various levels of factor <math>{{x}_{1}}</math> . Assuming there are no interactions between the reactor type and <math>{{x}_{1}}</math> , a regression model can be fitted to this data as shown next.
Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let <math>{{x}_{2}}</math> be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

::<math>{{x}_{2}}=\{\begin{array}{*{35}{l}}
0\text{ Reactor Type I} \\
1\text{ Reactor Type II} \\
\end{array}</math>

[[Image:doet5.3.png|thumb|center|400px|Yield data from the two types of reactors for a chemical process.]]

Data entry in DOE++ for this example is shown in Figure IndiVarDesignSshot. The regression model for this data is:

::<math>Y={{\beta }_{0}}+{{\beta }_{1}}{{x}_{1}}+{{\beta }_{2}}{{x}_{2}}+\epsilon </math>

The <math>X</math> and <math>y</math> matrices for the given data are:

[[Image:doe5.25.png|thumb|center|400px|Data from Table 5.3 as entered in DOE++.]]

The estimated regression coefficients for the model can be obtained using Eqn. (LeastSquareEstimate) as:

::<math>\begin{align}
& \hat{\beta }= & {{({{X}^{\prime }}X)}^{-1}}{{X}^{\prime }}y \\
& = & \left[ \begin{matrix}
153.7 \\
2.4 \\
-27.5 \\
\end{matrix} \right]
\end{align}</math>

Therefore, the fitted regression model is:

::<math>\hat{y}=153.7+2.4{{x}_{1}}-27.5{{x}_{2}}</math>

Note that since <math>{{x}_{2}}</math> represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against <math>{{x}_{1}}</math> and <math>{{x}_{2}}</math> in a two dimensional space (because the resulting surface plot will be meaningless for the dimension in <math>{{x}_{2}}</math> ). To illustrate this, a scatter plot of the data in Table 5.3 against <math>{{x}_{2}}</math> is shown in Figure IndiVarScatterPlot. It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial <math>F</math> test of Section 5.FtestPartial (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table of Figure IndiVarResultsSshot. The results show that <math>{{x}_{2}}</math> (reactor type) contributes significantly to the fitted regression model.

===Multicollinearity===

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables.
Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the <math>t</math> test on the individual coefficient or looking at the <math>p</math> value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the <math>F</math> test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

[[Image:doe5.26.png|thumb|center|400px|Scatter plot of the observed yield values in Table 5.3 against <math>x_2 </math> (reactor type)]]

[[Image:doe5.27.png|thumb|center|400px|DOE++ results for the data in Table 5.3.]]

Multicollinearity can be detected using the variance inflation factor (abbreviated <math>VIF</math> ). <math>VIF</math> for a coefficient <math>{{\beta }_{j}}</math> is defined as:

<math>VIF=\frac{1}{(1-R_{j}^{2})}</math>

where <math>R_{j}^{2}</math> is the coefficient of multiple determination resulting from regressing the <math>j</math> th predictor variable, <math>{{x}_{j}}</math> , on the remaining <math>k</math> -1 predictor variables. Mean values of <math>VIF</math> considerably greater than 1 indicate multicollinearity problems.
A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

'''Example 8'''

Variance inflation factors can be obtained for the data in Table 5.1. To calculate the variance inflation factor for <math>{{x}_{1}}</math> , <math>R_{1}^{2}</math> has to be calculated. <math>R_{1}^{2}</math> is the coefficient of determination for the model when <math>{{x}_{1}}</math> is regressed on the remaining variables. In the case of this example there is just one remaining variable which is <math>{{x}_{2}}</math> . If a regression model is fit to the data, taking <math>{{x}_{1}}</math> as the response variable and <math>{{x}_{2}}</math> as the predictor variable, then the design matrix and the vector of observations are:

::<math>{{X}_{{{R}_{1}}}}=\left[ \begin{matrix}
1 & 29.1 \\
1 & 29.3 \\
. & . \\
. & . \\
. & . \\
1 & 32.9 \\
\end{matrix} \right]\text{ }{{y}_{{{R}_{1}}}}=\left[ \begin{matrix}
41.9 \\
43.4 \\
. \\
. \\
. \\
77.8 \\
\end{matrix} \right]</math>

The regression sum of squares for this model can be obtained using Eqn. (RegressionSumofSquares) as:

::<math>\begin{align}
& S{{S}_{R}}= & y_{{{R}_{1}}}^{\prime }\left[ {{H}_{{{R}_{1}}}}-(\frac{1}{n})J \right]{{y}_{{{R}_{1}}}} \\
& = & 1988.6
\end{align}</math>

where <math>{{H}_{{{R}_{1}}}}</math> is the hat matrix (and is calculated using <math>{{H}_{{{R}_{1}}}}={{X}_{{{R}_{1}}}}{{(X_{{{R}_{1}}}^{\prime }{{X}_{{{R}_{1}}}})}^{-1}}X_{{{R}_{1}}}^{\prime }</math> ) and <math>J</math> is the matrix of ones. The total sum of squares for the model can be calculated using Eqn. (TotalSumofSquares) as:

::<math>\begin{align}
& S{{S}_{T}}= & {{y}^{\prime }}\left[ I-(\frac{1}{n})J \right]y \\
& = & 2182.9
\end{align}</math>

where <math>I</math> is the identity matrix. Therefore:

::<math>\begin{align}
& R_{1}^{2}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{1988.6}{2182.9} \\
& = & 0.911
\end{align}</math>

Then the variance inflation factor for <math>{{x}_{1}}</math> is:

::<math>\begin{align}
& VI{{F}_{1}}= & \frac{1}{(1-R_{1}^{2})} \\
& = & \frac{1}{1-0.911} \\
& = & 11.2
\end{align}</math>

The variance inflation factor for <math>{{x}_{2}}</math> , <math>VI{{F}_{2}}</math> , can be obtained in a similar manner. In DOE++, the variance inflation factors are displayed in the VIF column of the Regression Information Table as shown in Figure VIFSshot. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data in Table 5.1.

[[Image:doe5.28.png|thumb|center|400px|Variance inflation factors for the data in Table 5.1.]]

Simple Linear Regression Analysis

2012-08-23T06:04:49Z

Pengying niu: /* Lack-of-Fit Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
{{{\hat{\beta }}}_{1}} &= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \end{align}</math>
::<math>\begin{align}
{{{\hat{\beta }}}_{0}} &= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
S{{S}_{T}} & = & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \end{align} </math>
::<math>\begin{align}
S{{S}_{R}} &= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \end{align}
</math>
::<math>\begin{align}
S{{S}_{E}} &= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37 \end{align}
</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
S{{S}_{PE}} & = & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
dof(S{{S}_{PE}}) & = & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
M{{S}_{PE}} & = & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-23T06:03:13Z

Pengying niu: /* Lack-of-Fit Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
{{{\hat{\beta }}}_{1}} &= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \end{align}</math>
::<math>\begin{align}
{{{\hat{\beta }}}_{0}} &= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
S{{S}_{T}} & = & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \end{align} </math>
::<math>\begin{align}
S{{S}_{R}} &= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37 \end{align}
</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
S{{S}_{PE}} & = & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
dof(S{{S}_{PE}}) & = & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
M{{S}_{PE}} & = & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-23T05:58:46Z

Pengying niu: /* Lack-of-Fit Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
{{{\hat{\beta }}}_{1}} &= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \end{align}</math>
::<math>\begin{align}
{{{\hat{\beta }}}_{0}} &= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
S{{S}_{T}} & = & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
S{{S}_{PE}} & = & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
dof(S{{S}_{PE}}) & = & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
M{{S}_{PE}} & = & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-23T05:52:06Z

Pengying niu: /* Lack-of-Fit Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
{{{\hat{\beta }}}_{1}} &= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
S{{S}_{T}} & = & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
S{{S}_{PE}} & = & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
dof(S{{S}_{PE}}) & = & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
M{{S}_{PE}} & = & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-23T05:50:36Z

Pengying niu: /* Lack-of-Fit Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
{{{\hat{\beta }}}_{1}} &= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
& S{{S}_{PE}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{PE}})= & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
& M{{S}_{PE}}= & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-22T23:47:10Z

Pengying niu: /* Fitted Regression Line */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
& {{{\hat{\beta }}}_{1}}= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
& S{{S}_{PE}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{PE}})= & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
& M{{S}_{PE}}= & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-22T03:43:19Z

Pengying niu: /* Confidence Interval on New Observations */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 </math>=<math>\frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
{{{\hat{y}}}_{p}}&=& {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
& {{{\hat{\beta }}}_{1}}= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
& S{{S}_{PE}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{PE}})= & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
& M{{S}_{PE}}= & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-22T03:42:49Z

Pengying niu: /* F Test */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 </math>=<math>\frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{T}}&=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
&=& 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{R}} &=& \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
&=& 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
S{{S}_{E}}&= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
&=& 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
{{f}_{0}}&=& \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
&=& \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
&=& \frac{22607.81/1}{371.63/(25-2)} \\
&=& 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
p\text{ }value &=& 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
& {{{\hat{\beta }}}_{1}}= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
& S{{S}_{PE}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{PE}})= & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
& M{{S}_{PE}}= & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]

Simple Linear Regression Analysis

2012-08-22T00:11:08Z

Pengying niu: /* t Tests */

{{Template:Doebook|3}}

==Introduction==

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in DOE++ includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in our "Analysis of Experiments" discussion), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all DOE++ calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in the last section of Chapter 6, Use of Regression to Calculate Sum of Squares. Additionally, DOE++ also includes a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them. This chapter discusses simple linear regression analysis while Chapter 5 focuses on multiple linear regression analysis.

==Simple Linear Regression Analysis==

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see Table 4.1). This data can be entered in DOE++ as shown in Figure 4.1 and a scatter plot can be obtained as shown in Figure 4.2. [Note] In the scatter plot yield, <math>y_i </math> is plotted for different temperature values, <math>x_i </math> . It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables <math>x </math> and <math>Y</math>. [Note] However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between <math>x </math> and <math>Y </math> may be expressed as follows:
(1)
::<math>Y=\beta_0+\beta_1{x}+\epsilon </math>

[[Image:doet4.1.png|thumb|center|300px|Yield data observations of a chemical process at different values of reaction temperature.]]

[[Image:doe4.1.png|thumb|center|300px|Data entry in DOE++ for the observations in Table 4.1.]]

[[Image:doe4.2.png|thumb|center|300px|Scatter plot for the data in Table 4.1.]]

Eqn. (1) is the linear regression model that can be used to explain the relation between <math>x </math> and <math>Y </math> that is seen on the scatter plot above. In this model, the mean value of <math>Y </math> (abbreviated as <math>E(Y) </math>) is assumed to follow the linear relation <math>\beta_0=\beta_1{x} </math>:

::<math>E(Y)=\beta_0+\beta_1{x} </math>

The actual values of <math>Y </math>, (which are observed as yield from the chemical process from time to time and are random in nature), are assumed to be the sum of the mean value, <math>E(Y) </math> , and a random error term, <math>\epsilon </math> :

::<math>Y=E(Y)+\epsilon </math>
::<math>=\beta_0=\beta_1{x}+\epsilon </math>

The regression model here is called a ''simple'' linear regression model because there is just one independent variable, <math>x </math> , in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, <math>Y </math> , is also referred to as the response. The slope, <math>\beta_1 </math> , and the intercept, <math>\beta_0 </math> , of the line <math>E(Y)=\beta_0=\beta_1{x} </math> are called regression coefficients. The slope, <math>\beta_1 </math> , can be interpreted as the change in the mean value of <math>Y </math> for a unit change in <math>x </math>.

The random error term, <math>\epsilon </math> , is assumed to follow the normal distribution with a mean of 0 and variance of <math>\sigma^2 </math>. Since <math>Y </math> is the sum of this random term and the mean value, <math>E(Y)</math> , (which is a constant), the variance of <math>Y </math>at any given value of <math>x </math> is also <math>\sigma^2 </math>. Therefore, at any given value of <math>x </math>, say <math>x_i </math>, the dependent variable <math>Y </math> follows a normal distribution with a mean of <math>\beta_0+\beta_1{x_i} </math> and a standard deviation of <math>\sigma^2 </math>. This is illustrated in the following figure.

[[Image:doe4.3.png|thumb|center|400px|The normal distribution of for two values of . Also shown is the true regression line and the values of the random error term, , corresponding to the two values. The true regression line and are usually not known.]]

===Fitted Regression Line===
The true regression line corresponding to Eqn. (1) is usually never known. However, the regression line can be estimated by estimating the coefficients <math>\beta_1 </math> and <math>\beta_0 </math> for an observed data set. The estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are calculated using least squares. (For details on least square estimates refer to [19]). The estimated regression line, obtained using the values of <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, is called the fitted line. The least square estimates, <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math>, are obtained using the following equations:(2)

::<math>\widehat{\beta}_1 </math>=<math>\frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>\widehat{\beta}_0=\bar{y}-\widehat{\beta}_1 \bar{x} </math>
(3)

where <math>\bar{y} </math> is the mean of all the observed values and <math>\bar{x} </math> is the mean of all values of the predictor variable at which the observations were taken. <math>\bar{y} </math> is calculated using <math>\bar{y}=(1/n)\sum)_{i=1}^n y_i </math> and <math>\bar{x}=(1/n)\sum)_{i=1}^n x_i </math> is calculated using .

Once <math>\widehat{\beta}_1 </math> and <math>\widehat{\beta}_0 </math> are known, the fitted regression line can be written as:
(4)
::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>

where <math>\widehat{y} </math> is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, <math>E(Y) </math>. The fitted value,<math>\widehat{y}_i </math> , for a given value of the predictor variable, <math>x_i </math> , may be different from the corresponding observed value, <math>y_i </math>. The difference between the two values is called the residual, <math>e_i </math>: (5)

::<math>e_i=y_i-\widehat{y}_i </math>

====Calculation of the Fitted Line Using Least Square Estimates====
The least square estimates of the regression coefficients can be obtained for the data in Table 4.1 using the Eqns. (2) and (3) as follows:

::<math>\widehat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i- \frac{(\sum_{i=1}^n y_i) (\sum_{i=1}^n x_i)}{n}}{\sum_{i=1}^n (x_i-\bar{x})^2} </math>
::<math>=\frac{322516-\frac{4158 x 1871}{25}}{5697.36} </math>
::<math>=1.9952 \approx 2.00 </math>

::<math>\widehat{\beta}_0 = \bar{y}-\widehat{\beta}_1 \bar{x} </math>
::<math>= 166.32 - 2 </math> x <math>74.84 </math>
::<math>=17.0016 \approx 17.00</math>

Knowing <math>\widehat{\beta}_0 </math> and <math>\widehat{\beta}_1 </math>, the fitted regression line is:

::<math>\widehat{y}=\widehat{\beta}_0+\widehat{\beta}_1 x </math>
:: <math>= 17.0016+1.9952 </math> x <math>x </math>
::<math>\approx 17+2 x </math>

This line is shown in Figure 4.4.

[[Image:doe4.4.png|thumb|center|400px|Fitted regression line for the data in Table 4.1. Also shown is the residual for the 21st observation.]]

Once the fitted regression line is known, the fitted value of <math>Y </math> corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in Table 4.1 is:

::<math>\widehat{y}_{21} = \widehat{\beta}_0 = \widehat{\beta}_1 x_{21} </math>
::<math>=(17.0016) + (1.9952) </math> x <math>93 </math>
::<math>=202.6 </math>

The observed response at this point is <math>y_{21}=194 </math>. Therefore, the residual at this point is:

::<math>e_{21}= y_{21}-\widehat{y}_{21} </math>
::<math>=194-202.6 </math>
::<math>=-8.6 </math>

In DOE++, fitted values and residuals are available using the Diagnostic icon in the Control Panel. The values are shown in Figure 4.5.

[[Image:doe4.5.png|thumb|center|400px|Fitted values and residuals for the data in Table 4.1.]]

==Hypothesis Tests in Simple Linear Regression==

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, <math>\epsilon </math> , is normally and independently distributed with a mean of zero and variance of <math>\sigma^2</math>.

===<math>t </math> Tests===

The tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the <math>t </math> distribution is used to test the two-sided hypothesis that the true slope, <math>\beta_1 </math> , equals some constant value, <math>\beta_{1,0} </math>. [Note] The statements for the hypothesis test are expressed as:

::<math>H_0</math> : <math>\beta_1 = \beta_{1,0} </math>
::<math>H_1</math> : <math>\beta_{1}\ne\beta_{1,0} </math>

The test statistic used for this test is:

::<math>T_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_1)} </math>

where <math>\widehat{\beta}_1 </math> is the least square estimate of <math>\beta_1 </math>, and <math>se(\widehat{\beta}_1) </math> is its standard error. The value of <math>se(\widehat{\beta}_1) </math> can be calculated as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>

The test statistic, <math>T_0 </math> , follows a <math>t </math> distribution with <math>(n-2) </math> degrees of freedom, where <math>n </math> is the total number of observations. The null hypothesis, <math>H_0 </math>, is rejected if the calculated value of the test statistic is such that:

::<math>-t_{\alpha/2,n-2}<T_0<t_{\alpha/2,n-2} </math>

where <math>t_{\alpha/2,n-2} </math> and <math>-t_{\alpha/2,n-2} </math> are the critical values for the two-sided hypothesis. <math>t_{\alpha/2,n-2} </math> is the percentile of the <math>t </math> distribution corresponding to a cumulative probability of (<math>(1-\alpha/2) </math>) and <math>\alpha </math> is the significance level.

If the value of <math>\beta_{1,0} </math> used in Eqn. (6) is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between <math>x </math> and <math>Y </math>. Failure to reject <math>H_0:\beta_1=0 </math> implies that no linear relationship exists between <math>x </math> and <math>Y </math>. This result may be obtained when the scatter plots of against are as shown in 4.6 (a) and (b) of the following figure. Figure 4.6 (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. Figure 4.6 (b) represents the case where the true relationship between <math>x </math> and <math>Y </math> is not linear. Figure 4.6 (c) and (d) represent the case when <math>H_0:\beta_1=0 </math> is rejected, implying that a model does exist between <math>x </math> and <math>Y </math>. Figure 4.6 (c) represents the case where the linear model is sufficient. Figure 4.6, (d) represents the case where a higher order model may be needed.

[[Image:doe4.6.png|thumb|center|400px|Possible scatter plots of against . Plots (a) and (b) represent cases when is not rejected. Plots (c) and (d) represent cases when is rejected.]]

A similar procedure can be used to test the hypothesis on the intercept . The test statistic used in this case is:

::<math>T_0=\frac{\widehat{\beta}_0-\beta_{0,0}}{se(\widehat{\beta}_0)} </math>(8)

where is the least square estimate of , and is its standard error which is calculated using:

::<math>se(\widehat{\beta}_0)= \sqrt{\frac{\displaystyle\sum_{i=1}^n e_i^2}{n-2} \Bigg[ \frac{1}{n}+\frac{\bar{x}^2}{\displaystyle\sum_{i=1}^n (x_i-\bar{x})^2} \Bigg]} </math>
(9)

'''Example 4.1'''

The test for the significance of regression for the data in Table 4.1 is illustrated in this example. The test is carried out using the <math>t </math> test on the coefficient <math>\beta_1 </math>. The hypothesis to be tested is <math>H_0 : \beta_1 = 0 </math>. To calculate the statistic to test <math>H_0 </math>, the estimate, <math>\widehat{\beta}_1 </math>, and the standard error, <math>se(\widehat{\beta}_1) </math>, are needed. The value of <math>\widehat{\beta}_1 </math> was obtained in Chapter 4, Fitted Regression Line. The standard error can be calculated using Eqn. (7) as follows:

::<math>se(\widehat{\beta}_1)= \sqrt{\frac{\frac{\displaystyle \sum_{i=1}^n e_i^2}{n-2}}{\displaystyle \sum_{i=1}^n (x_i-\bar{x})^2}} </math>
::<math> = \sqrt{\frac{(371.627/23)}{5679.36}} </math>
::<math> = 0.0533 </math>

Then, the test statistic can be calculated using the following equation:

::<math>t_0=\frac{\widehat{\beta}_1-\beta_{1,0}}{se(\widehat{\beta}_0)} </math>
::<math>=\frac{1.9952-0}{0.0533} </math>
::<math>=37.4058 </math>

The <math>p </math> value corresponding to this statistic based on the <math>t </math> distribution with 23(<math>n-2=25-2=23 </math>) degrees of freedom can be obtained as follows:

::<math>p value = 2\times (1-P(T\le t_0) </math>
::<math>= 2 \times (1-0.999999) </math>
::<math>= 0</math>

Assuming that the desired significance level is 0.1, since <math>p </math> value < 0.1, <math>H_0 : \beta_1=0 </math> is rejected indicating that a relation exists between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure 4.2, it can be concluded that the relationship between temperature and yield is linear.

In DOE++, information related to the <math>t </math> test is displayed in the Regression Information table as shown in Figure 4.7. In this table the <math>t </math> test for <math>\beta_1 </math> is displayed in the row for the term Temperature because <math>\beta_1 </math> is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the <math>p </math> value for the <math>t </math> test, respectively. These values have been calculated for <math>\beta_1 </math> in this example. The Coefficient column represents the estimate of regression coefficients. For <math>\beta_1 </math>, this value was calculated using Eqn. (2). The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Chapter 7, Two Level Factorial Experiments. Columns Low CI and High CI represent the limits of the confidence intervals for the regression coefficients and are explained in Chapter 4, Confidence Interval on Regression Coefficients. The Variance Inflation Factor column displays values that give a measure of multicollinearity. The concept of multicollinearity is only applicable to multiple linear regression models and is explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.7.png|thumb|center|400px|Regression results for the data in Table 4.1.]]

===Analysis of Variance Approach to Test the Significance of Regression===

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

====Sum of Squares====

The total variance (i.e. the variance of all of the observed data) is estimated using the observed data. As mentioned in Chapter 3, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

::<math>{{s}^{2}}=\frac{\underset{i=1}{\overset{n}{\mathop{\sum }}}\,{{({{y}_{i}}-\bar{y})}^{2}}}{n-1}</math>

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, <math>{{y}_{i}}</math> , from their mean, <math>\bar{y}</math> . In the context of ANOVA this quantity is called the total sum of squares (abbreviated <math>S{{S}_{T}}</math> ) because it relates to the total variance of the observations. Thus:

::<math>S{{S}_{T}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}</math>

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with <math>S{{S}_{T}}</math> , <math>dof(S{{S}_{T}})</math> , is <math>n-1</math> . The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated <math>M{{S}_{T}}</math> ) is:

::<math>M{{S}_{T}}=\frac{S{{S}_{T}}}{dof(S{{S}_{T}})}=\frac{S{{S}_{T}}}{n-1}</math>

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see Figure PerfectModel (a)). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated <math>S{{S}_{R}}</math> ) equals the total sum of squares; i.e. the model explains all of the observed variance:

::<math>S{{S}_{R}}=S{{S}_{T}}</math>

For the perfect model, the regression sum of squares, <math>S{{S}_{R}}</math> , equals the total sum of squares, <math>S{{S}_{T}}</math> , because all estimated values, <math>{{\hat{y}}_{i}}</math> , will equal the corresponding observations, <math>{{y}_{i}}</math> . <math>S{{S}_{R}}</math> can be calculated using a relationship similar to the one for obtaining <math>S{{S}_{T}}</math> by replacing <math>{{y}_{i}}</math> by <math>{{\hat{y}}_{i}}</math> in the relationship of <math>S{{S}_{T}}</math> . Therefore:

::<math>S{{S}_{R}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{R}}</math> , <math>dof(S{{S}_{R}})</math> , is one.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in Figure PerfectModel (b) or Figure FittedRegressionLine.

[[Image:doe4.8.png|thumb|center|400px|A perfect regression model will pass through all observed data points as shown in *(a). Most models are imperfect and do not fit perfectly to all data points as shown in (b).]]

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated <math>S{{S}_{E}}</math> ). The deviation for this sum of squares is obtained at each observation in the form of the residuals, <math>{{e}_{i}}</math> . The error sum of squares can be obtained as the sum of squares of these deviations:

::<math>S{{S}_{E}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,e_{i}^{2}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

The number of degrees of freedom associated with <math>S{{S}_{E}}</math> , <math>dof(S{{S}_{E}})</math> , is <math>(n-2)</math> .
The total variability of the observed data (i.e. total sum of squares, <math>S{{S}_{T}}</math> ) can be written using the portion of the variability explained by the model, <math>S{{S}_{R}}</math> , and the portion unexplained by the model, <math>S{{S}_{E}}</math> , as:

::<math>S{{S}_{T}}=S{{S}_{R}}+S{{S}_{E}}</math>

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

::<math>\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{\hat{y}}_{i}}-\bar{y})}^{2}}+\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{\hat{y}}_{i}})}^{2}}</math>

[[Image:doe4.9.png|thumb|center|400px|Scatter plots showing the deviations for the sum of squares used in ANOVA. (a) shows deviations for , (b) shows deviations for , and (c) shows deviations for .]]

====Mean Squares====

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, <math>M{{S}_{E}}</math> , can be obtained as:

::<math>M{{S}_{E}}=\frac{S{{S}_{E}}}{dof(S{{S}_{E}})}=\frac{S{{S}_{E}}}{n-2}</math>

The error mean square is an estimate of the variance, <math>{{\sigma }^{2}}</math> , of the random error term, <math>\epsilon </math> , and can be written as:

::<math>{{\hat{\sigma }}^{2}}=\frac{S{{S}_{E}}}{n-2}</math>

Similarly, the regression mean square, <math>M{{S}_{R}}</math> , can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

::<math>M{{S}_{R}}=\frac{S{{S}_{R}}}{dof(S{{S}_{R}})}=\frac{S{{S}_{R}}}{1}</math>

====F Test====

To test the hypothesis <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> , the statistic used is based on the <math>F</math> distribution. It can be shown that if the null hypothesis <math>{{H}_{0}}</math> is true, then the statistic:

::<math>{{F}_{0}}=\frac{M{{S}_{R}}}{M{{S}_{E}}}=\frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)}</math>

follows the <math>F</math> distribution with <math>1</math> degree of freedom in the numerator and <math>(n-2)</math> degrees of freedom in the denominator. <math>{{H}_{0}}</math> is rejected if the calculated statistic, <math>{{F}_{0}}</math> , is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,1,n-2}}</math>

where <math>{{f}_{\alpha ,1,n-2}}</math> is the percentile of the <math>F</math> distribution corresponding to a cumulative probability of ( <math>1-\alpha </math> ) and <math>\alpha </math> is the significance level.

'''Example 2'''

The analysis of variance approach to test the significance of regression can be applied to the yield data in Table 4.1. To calculate the statistic, <math>{{F}_{0}}</math> , for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next.
The total sum of squares can be calculated as:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-166.32)}^{2}} \\
& = & 22979.44
\end{align}</math>

The regression sum of squares can be calculated as:

::<math>\begin{align}
& S{{S}_{R}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-166.32)}^{2}} \\
& = & 22607.81
\end{align}</math>

The error sum of squares can be calculated as:

::<math>\begin{align}
& S{{S}_{E}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 371.63
\end{align}</math>

Knowing the sum of squares, the statistic to test <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> can be calculated as follows:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{R}}}{M{{S}_{E}}} \\
& = & \frac{S{{S}_{R}}/1}{S{{S}_{E}}/(n-2)} \\
& = & \frac{22607.81/1}{371.63/(25-2)} \\
& = & 1399.20
\end{align}</math>

The critical value at a significance level of 0.1 is <math>{{f}_{0.05,1,23}}=2.937</math> . Since <math>{{f}_{0}}>{{f}_{\alpha ,1,n-2}},</math> <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected and it is concluded that <math>{{\beta }_{1}}</math> is not zero. Alternatively, the <math>p</math> value can also be used. The <math>p</math> value corresponding to the test statistic, <math>{{f}_{0}}</math> , based on the <math>F</math> distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.999999 \\
& = & 4.17E-22
\end{align}</math>

Assuming that the desired significance is 0.1, since the <math>p</math> value < 0.1, then <math>{{H}_{0}}\ \ :\ \ {{\beta }_{1}}=0</math> is rejected, implying that a relation does exist between temperature and yield for the data in Table 4.1. Using this result along with the scatter plot of Figure ScatterPlotSshot, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in Figure Ex2ANOVAtableSshot. Note that this is the same result that was obtained from the <math>t</math> test in Section 4.tTest. The ANOVA and Regression Information tables in DOE++ represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Chapter 5. The term Partial appearing in Figure Ex2ANOVAtableSshot relates to the extra sum of squares and is also explained in Chapter 5.

[[Image:doe4.10.png|thumb|center|400px|ANOVA table for the data in Table 4.1.]]

==Confidence Intervals in Simple Linear Regression==

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of <math>A</math> and an upper limit of <math>B</math> implies that 90% of the population lies between the values of <math>A</math> and <math>B</math> . Out of the remaining 10% of the population, 5% is less than <math>A</math> and 5% is greater than <math>B</math> . (For details refer to [LDAReference]). This section discusses confidence intervals used in simple linear regression analysis.

===Confidence Interval on Regression Coefficients===

A 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{1}}</math> is obtained as follows:

::<math>{{\hat{\beta }}_{1}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{1}})</math>

Similarly, a 100( <math>1-\alpha </math> ) percent confidence interval on <math>{{\beta }_{0}}</math> is obtained as:

::<math>{{\hat{\beta }}_{0}}\pm {{t}_{\alpha /2,n-2}}\cdot se({{\hat{\beta }}_{0}})</math>

===Confidence Interval on Fitted Values===

A 100( <math>1-\alpha </math> ) percent confidence interval on any fitted value, <math>{{\hat{y}}_{i}}</math> , is obtained as follows:

::<math>{{\hat{y}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]}</math>

It can be seen that the width of the confidence interval depends on the value of <math>{{x}_{i}}</math> and will be a minimum at <math>{{x}_{i}}=\bar{x}</math> and will widen as <math>\left| {{x}_{i}}-\bar{x} \right|</math> increases.

===Confidence Interval on New Observations===

For the data in Table 4.1, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If <math>{{x}_{p}}</math> is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times {{x}_{p}}
\end{align}</math>

If a confidence interval needs to be obtained on <math>{{\hat{y}}_{p}}</math> , then this interval should include both the error from the fitted model and the error associated with future observations. This is because <math>{{\hat{y}}_{p}}</math> represents the estimate for a value of <math>Y</math> that was not used to obtain the regression model. The confidence interval on <math>{{\hat{y}}_{p}}</math> is referred to as the prediction interval <math>.</math> A 100( <math>1-\alpha </math> ) percent prediction interval on a new observation is obtained as follows:

::<math>{{\hat{y}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]}</math>

'''Example 3'''

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at <math>x=93</math> for the data in Table 4.1 is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at <math>x=91</math> .

The fitted value, <math>{{\hat{y}}_{i}}</math> , corresponding to <math>x=93</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{21}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{21}} \\
& = & 17.0016+1.9952\times 93 \\
& = & 202.6
\end{align}</math>

The 95% confidence interval <math>(\alpha =0.05)</math> on the fitted value, <math>{{\hat{y}}_{21}}=202.6</math> , is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{i}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ \frac{1}{n}+\frac{{{({{x}_{i}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \right]} \\
& = & 202.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.069\sqrt{16.16\left[ \frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 202.6\pm 2.602
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{21}}</math> are 199.95 and 205.2, respectively.
The estimated value based on the fitted regression model for the new observation at <math>x=91</math> is:

::<math>\begin{align}
& {{{\hat{y}}}_{p}}= & {{{\hat{\beta }}}_{0}}+{{{\hat{\beta }}}_{1}}{{x}_{p}} \\
& = & 17.0016+1.9952\times 91 \\
& = & 198.6
\end{align}</math>

The 95% prediction interval on <math>{{\hat{y}}_{p}}=198.6</math> is:

::<math>\begin{align}
& = & {{{\hat{y}}}_{p}}\pm {{t}_{\alpha /2,n-2}}\sqrt{{{{\hat{\sigma }}}^{2}}\left[ 1+\frac{1}{n}+\frac{{{({{x}_{p}}-\bar{x})}^{2}}}{\underset{i=1}{\overset{n}{\mathop \sum }}\,{{({{x}_{p}}-\bar{x})}^{2}}} \right]} \\
& = & 198.6\pm {{t}_{0.025,23}}\sqrt{M{{S}_{E}}\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\sqrt{16.16\left[ 1+\frac{1}{25}+\frac{{{(93-74.84)}^{2}}}{5679.36} \right]} \\
& = & 198.6\pm 2.069\times 4.1889 \\
& = & 198.6\pm 8.67
\end{align}</math>

The 95% limits on <math>{{\hat{y}}_{p}}</math> are 189.9 and 207.2, respectively. In DOE++, confidence and prediction intervals are available using the Prediction icon in the Control Panel. The prediction interval values calculated in this example are shown in Figure PredictionInterval as Low PI and High PI respectively. The columns labeled Mean Predicted and Standard Error represent the values of <math>{{\hat{y}}_{p}}</math> and the standard error used in the calculations.

[[Image:doe4.11.png|thumb|center|400px|Calculation of prediction intervals in DOE++.]]

==Measures of Model Adequacy==

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

===Coefficient of Determination (<math>R^2 </math>)===
The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, . The amount of this variability explained by the regression model is the regression sum of squares, . The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

::<math>R^2 = \frac{SS_R}{SS_T} </math>(22)

<math>R^2 </math> can take on values between 0 and 1 since <math>R^2 = \frac{SS_R}{SS_T} </math> . For the yield data example, <math>R^2 </math> can be calculated as:

::<math>\begin{align}
& {{R}^{2}}= & \frac{S{{S}_{R}}}{S{{S}_{T}}} \\
& = & \frac{22607.81}{22979.44} \\
& = & 0.98
\end{align}</math>

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of <math>{{R}^{2}}</math> indicate a better fitting regression model. However, <math>{{R}^{2}}</math> should be used cautiously as this is not always the case. The value of <math>{{R}^{2}}</math> increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of <math>{{R}^{2}}</math> cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, <math>M{{S}_{E}}</math> , for the new model is larger than the <math>M{{S}_{E}}</math> of the older model, even though the new model will show an increased value of <math>{{R}^{2}}</math> . In the results obtained from DOE++, <math>{{R}^{2}}</math> is displayed as R-sq under the ANOVA table (as shown in Figure FullAnalysisEx1 which displays the complete analysis sheet for the data in Table 4.1).
The other values displayed with <math>{{R}^{2}}</math> are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, <math>M{{S}_{E}}</math> , and represents Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of indicate a better fitting regression model. However, should be used cautiously as this is not always the case. The value of increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, , for the new model is larger than the of the older model, even though the new model will show an increased value of . In the results obtained from DOE++, is displayed as R-sq under the ANOVA table (as shown in Figure 4.12, which displays the complete analysis sheet for the data in Table 4.1).

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, , and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Chapter 5, Multiple Linear Regression Analysis.

[[Image:doe4.12.png|thumb|center|400px|Complete analysis for the data in Table 4.1.]]

===Residual Analysis===
In the simple linear regression model the true error terms, <math>{{\epsilon }_{i}}</math> , are never known. The residuals, <math>{{e}_{i}}</math> , may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, <math>{{\epsilon }_{i}},</math> are assumed to be normally distributed with a mean of zero and a variance of <math>{{\sigma }^{2}}</math> , in a good model the observed error terms, (i.e. the residuals, <math>{{e}_{i}}</math> ,) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of <math>{{\sigma }^{2}}</math> . Residuals are usually plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , against the predictor variable values, <math>{{x}_{i}}</math> , and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

:1. Residuals follow the normal distribution.
:2. Residuals have a constant variance.
:3. Regression function is linear.
:4. A pattern does not exist when residuals are plotted in a time or run-order sequence.
:5. There are no outliers.

Examples of residual plots are shown in Figure DiffrResidualPlots. The plot of Figure DiffrResidualPlots (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. The plot of Figure DiffrResidualPlots (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on <math>Y</math> may be helpful in this case (see Section 4.Transformations). If the residuals follow the pattern of Figure DiffrResidualPlots (c) or (d) then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on <math>x</math> or <math>Y</math> may be required in such cases. A plot of residuals may also show a pattern as seen in Figure DiffrResidualPlots (e) indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

[[Image:doe4.13.png|thumb|center|300px|Possible residual plots (against fitted values, time or run-order) that can be obtained from simple linear regression analysis.]]

'''Example 4.4'''

Residual plots for the data of Table 4.1 are shown in Figures ResidualNPP to ResidualVsRun. Figure ResidualNPP is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In Figure ResidualVsFitted the residuals are plotted against the fitted values, <math>{{\hat{y}}_{i}}</math> , and in Figure ResidualVsRun the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this oulier.

[[Image:doe4.14.png|thumb|center|300px|Normal probability plot of residuals for the data in Table 4.1.]]

[[Image:doe4.15.png|thumb|center|300px|Plot of residuals against fitted values for the data in Table 4.1.]]

[[Image:doe4.16.png|thumb|center|300px|Plot of residuals against run order for the data in Table 4.1.]]
 

===Lack-of-Fit Test===

As mentioned in Section 4.ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ( <math>S{{S}_{E}}=0</math> ). Thus, no error exists for the perfect model. However, if you record the response values for the same values of <math>{{x}_{i}}</math> for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated <math>S{{S}_{PE}}</math> ) quantifies these variations. <math>S{{S}_{PE}}</math> is calculated by taking repeated observations at some or all values of <math>{{x}_{i}}</math> and adding up the square of deviations at each level of <math>x</math> using the respective repeated observations at that <math>x</math> value.
Assume that there are <math>n</math> levels of <math>x</math> and <math>{{m}_{i}}</math> repeated observations are taken at each <math>i</math> th level. The data is collected as shown next:

::<math>\begin{align}
& & {{y}_{11}},{{y}_{12}},....,{{y}_{1{{m}_{1}}}}\text{ repeated observations at }{{x}_{1}} \\
& & {{y}_{21}},{{y}_{22}},....,{{y}_{2{{m}_{2}}}}\text{ repeated observations at }{{x}_{2}} \\
& & ... \\
& & {{y}_{i1}},{{y}_{i2}},....,{{y}_{i{{m}_{i}}}}\text{ repeated observations at }{{x}_{i}} \\
& & ... \\
& & {{y}_{n1}},{{y}_{n2}},....,{{y}_{n{{m}_{n}}}}\text{ repeated observations at }{{x}_{n}}
\end{align}</math>

The sum of squares of the deviations from the mean of the observations at <math>i</math> th level of <math>x</math> , <math>{{x}_{i}}</math> , can be calculated as:

::<math>\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

where <math>{{\bar{y}}_{i}}</math> is the mean of the <math>{{m}_{i}}</math> repeated observations corresponding to <math>{{x}_{i}}</math> ( <math>{{\bar{y}}_{i}}=(1/{{m}_{i}})\mathop{}_{j=1}^{{{m}_{i}}}{{y}_{ij}}</math> ). The number of degrees of freedom for these deviations is ( <math>{{m}_{i}}-1</math> ) as there are <math>{{m}_{i}}</math> observations at <math>i</math> th level of <math>x</math> but one degree of freedom is lost in calculating the mean, <math>{{\bar{y}}_{i}}</math> .

The total sum of square deviations (or <math>S{{S}_{PE}}</math> ) for all levels of <math>x</math> can be obtained by summing the deviations for all <math>{{x}_{i}}</math> as shown next:

::<math>S{{S}_{PE}}=\underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{\bar{y}}_{i}})}^{2}}</math>

The total number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,({{m}_{i}}-1) \\
& = & \underset{i=1}{\overset{n}{\mathop \sum }}\,{{m}_{i}}-n
\end{align}</math>

If all <math>{{m}_{i}}=m</math> , (i.e. <math>m</math> repeated observations are taken at all levels of <math>x</math> ), then <math>\mathop{}_{i=1}^{n}{{m}_{i}}=nm</math> and the degrees of freedom associated with <math>S{{S}_{PE}}</math> are:

::<math>=nm-n</math>

The corresponding mean square in this case will be:

::<math>M{{S}_{PE}}=\frac{S{{S}_{PE}}}{nm-n}</math>

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , is also considered as the error sum of squares, <math>S{{S}_{E}}</math> . For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, <math>S{{S}_{E}}</math> . One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated <math>S{{S}_{LOF}}</math> ) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

::<math>S{{S}_{E}}=S{{S}_{PE}}+S{{S}_{LOF}}</math>

Knowing <math>S{{S}_{E}}</math> and <math>S{{S}_{PE}}</math> , the previous equation can be used to obtain <math>S{{S}_{LOF}}</math> :

::<math>S{{S}_{LOF}}=S{{S}_{E}}-S{{S}_{PE}}</math>

The degrees of freedom associated with <math>S{{S}_{LOF}}</math> can be obtained in a similar manner using subtraction. For the case when <math>m</math> repeated observations are taken at all levels of <math>x</math> , the number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>dof(S{{S}_{PE}})=nm-n</math>

Since there are <math>nm</math> total observations, the number of degrees of freedom associated with <math>S{{S}_{E}}</math> is:

::<math>dof(S{{S}_{E}})=nm-2</math>

Therefore, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& = & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & n-2
\end{align}</math>

The corresponding mean square, <math>M{{S}_{LOF}}</math> , can now be obtained as:

::<math>M{{S}_{LOF}}=\frac{S{{S}_{LOF}}}{n-2}</math>

The magnitude of <math>S{{S}_{LOF}}</math> or <math>M{{S}_{LOF}}</math> will provide an indication of how far the regression model is from the perfect model. An <math>F</math> test exists to examine the lack-of-fit at a particular significance level. The quantity <math>M{{S}_{LOF}}/M{{S}_{PE}}</math> follows an <math>F</math> distribution with <math>(n-2)</math> degrees of freedom in the numerator and <math>(nm-n)</math> degrees of freedom in the denominator when all <math>{{m}_{i}}</math> equal <math>m</math> . The test statistic for the lack-of-fit test is:

::<math>{{F}_{0}}=\frac{M{{S}_{LOF}}}{M{{S}_{PE}}}</math>

If the critical value <math>{{f}_{\alpha ,n-2,mn-n}}</math> is such that:

::<math>{{F}_{0}}>{{f}_{\alpha ,n-2,nm-n}}</math>

it will lead to the rejection of the hypothesis that the model adequately fits the data.

'''Example 5'''

Assume that a second set of observations are taken for the yield data of Table 4.1. The resulting observations are recorded in Table 4.2. To conduct a lack-of-fit test on this data, the statistic <math>{{F}_{0}}=M{{S}_{LOF}}/M{{S}_{PE}}</math> , can be calculated as shown next.

[[Image:doet4.2.png|thumb|center|400px|Yield data from the first and second observation sets for the chemical process example in Section 4.1.]]

'''Calculation of Least Square Estimates'''

The parameters of the fitted regression model can be obtained using Eqns. (beta0) and (beta1) as:

::<math>\begin{align}
& {{{\hat{\beta }}}_{1}}= & \frac{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}}{{x}_{i}}-\frac{\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{y}_{i}} \right)\left( \underset{i=1}{\overset{50}{\mathop \sum }}\,{{x}_{i}} \right)}{50}}{\underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{x}_{i}}-\bar{x})}^{2}}} \\
& = & \frac{648532-\frac{8356\times 3742}{50}}{11358.72} \\
& = & 2.04 \\
& & \\
& {{{\hat{\beta }}}_{0}}= & \bar{y}-{{{\hat{\beta }}}_{1}}\bar{x} \\
& = & 167.12-2.04\times 74.84 \\
& = & 14.47
\end{align}</math>

Knowing <math>{{\hat{\beta }}_{1}}</math> and <math>{{\hat{\beta }}_{0}}</math> , the fitted values, <math>{{\hat{y}}_{i}}</math> , can be calculated.

'''Calculation of the Sum of Squares'''

Using the fitted values, the sum of squares can be obtained as follows:

::<math>\begin{align}
& S{{S}_{T}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-\bar{y})}^{2}} \\
& = & 47907.28 \\
& S{{S}_{R}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{{\hat{y}}}_{i}}-\bar{y})}^{2}} \\
& = & 47258.91 \\
& S{{S}_{E}}= & \underset{i=1}{\overset{50}{\mathop \sum }}\,{{({{y}_{i}}-{{{\hat{y}}}_{i}})}^{2}} \\
& = & 648.37
\end{align}</math>

'''Calculation of <math>M{{S}_{LOF}}</math>'''

The error sum of squares, <math>S{{S}_{E}}</math> , can now be split into the sum of squares due to pure error, <math>S{{S}_{PE}}</math> , and the sum of squares due to lack-of-fit, <math>S{{S}_{LOF}}</math> . <math>S{{S}_{PE}}</math> can be calculated as follows considering that in this example <math>n=25</math> and <math>m=2</math> :

::<math>\begin{align}
& S{{S}_{PE}}= & \underset{i=1}{\overset{n}{\mathop \sum }}\,\underset{j=1}{\overset{{{m}_{i}}}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & \underset{i=1}{\overset{25}{\mathop \sum }}\,\underset{j=1}{\overset{2}{\mathop \sum }}\,{{({{y}_{ij}}-{{{\bar{y}}}_{i}})}^{2}} \\
& = & 350
\end{align}</math>

The number of degrees of freedom associated with <math>S{{S}_{PE}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{PE}})= & nm-n \\
& = & 25\times 2-25 \\
& = & 25
\end{align}</math>

The corresponding mean square, <math>M{{S}_{PE}}</math> , can now be obtained as:

::<math>\begin{align}
& M{{S}_{PE}}= & \frac{S{{S}_{PE}}}{dof(S{{S}_{PE}})} \\
& = & \frac{350}{25} \\
& = & 14
\end{align}</math>

::<math>S{{S}_{LOF}}</math> can be obtained by subtraction from <math>S{{S}_{E}}</math> as:

::<math>\begin{align}
& S{{S}_{LOF}}= & S{{S}_{E}}-S{{S}_{PE}} \\
& = & 648.37-350 \\
& = & 298.37
\end{align}</math>

Similarly, the number of degrees of freedom associated with <math>S{{S}_{LOF}}</math> is:

::<math>\begin{align}
& dof(S{{S}_{LOF}})= & dof(S{{S}_{E}})-dof(S{{S}_{PE}}) \\
& = & (nm-2)-(nm-n) \\
& = & 23
\end{align}</math>

The lack-of-fit mean square is:

::<math>\begin{align}
& M{{S}_{LOF}}= & \frac{M{{S}_{LOF}}}{dof(M{{S}_{LOF}})} \\
& = & \frac{298.37}{23} \\
& = & 12.97
\end{align}</math>

'''Calculation of the Test Statistic'''

The test statistic for the lack-of-fit test can now be calculated as:

::<math>\begin{align}
& {{f}_{0}}= & \frac{M{{S}_{LOF}}}{M{{S}_{PE}}} \\
& = & \frac{12.97}{14} \\
& = & 0.93
\end{align}</math>

The critical value for this test is:

::<math>{{f}_{0.05,23,25}}=1.97</math>

Since <math>{{f}_{0}}<{{f}_{0.05,23,25}}</math> , we fail to reject the hypothesis that the model adequately fits the data. The <math>p</math> value for this case is:

::<math>\begin{align}
& p\text{ }value= & 1-P(F\le {{f}_{0}}) \\
& = & 1-0.43 \\
& = & 0.57
\end{align}</math>

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, <math>y=14.47+2.04x</math> , is adequate for the observed data. Table 4.3 presents a summary of the ANOVA calculations for the lack-of-fit test.

[[Image:doet4.3.png|thumb|center|500px|ANOVA table for the lack-of-fit test of the yield data example.]]

==Transformations==

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, <math>x</math> , or the response variable, <math>Y</math> , may often be sufficient to make the linear regression model appropriate for the transformed data.
If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on <math>Y</math> (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) might be useful. For data following the Poisson distribution, a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is generally applicable.

Transformations on <math>Y</math> may also be applied based on the type of scatter plot obtained from the data. Figure TransformationScatterPlots shows a few such examples. For the scatter plot of Figure (a), a square root transformation ( <math>{{Y}^{*}}=\sqrt{Y}</math> ) is applicable. While for Figure (b), a logarithmic transformation (i.e. <math>{{Y}^{*}}=\log (Y)</math> ) may be applied. For Figure (c), the reciprocal transformation ( <math>{{Y}^{*}}=1/Y</math> ) is applicable. At times it may be helpful to introduce a constant into the transformation of <math>Y</math> . For example, if <math>Y</math> is negative and the logarithmic transformation on <math>Y</math> seems applicable, a suitable constant, <math>k</math> , may be chosen to make all observed <math>Y</math> positive. Thus the transformation in this case would be <math>{{Y}^{*}}=\log (k+Y)</math> .
The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

::<math>{{Y}^{*}}={{Y}^{\lambda }}</math>

Here the parameter <math>\lambda </math> is determined using the given data such that <math>S{{S}_{E}}</math> is minimized (details on this method are presented in Chapter 6).

[[Image:doe4.17.png|thumb|center|400px|Transformations on for a few possible scatter plots. Plot (a) may require , (b) may require and (c) may require.]]