Report

Week 13 November 24-28 Three Mini-Lectures QMM 510 Fall 2014 ML 13.1 What Is Multicollinearity? • • • 13-2 Multicollinearity occurs when the independent variables X1, X2, …, Xm are intercorrelated instead of being independent. Collinearity occurs if only two predictors are correlated. The degree of multicollinearity is the real concern. Chapter 13 Multicollinearity Variance Inflation • • • 13-3 Multicollinearity induces variance inflation in the estimation of the regression model coefficients. This results in wider confidence intervals for the true coefficients b1, b2, …, bk and makes the t statistic less reliable. The separate contribution of each predictor in “explaining” the response variable is difficult to identify. Chapter 13 Multicollinearity Correlation Matrix To check whether two predictors are correlated (collinearity), inspect the correlation matrix using Excel, MegaStat, or MINITAB. For example, 13-4 Chapter 13 Multicollinearity Variance Inflation Factor (VIF) • The matrix scatter plots and correlation matrix only show correlations between any two predictors. • The variance inflation factor (VIF) is a more comprehensive test for multicollinearity. • For a given predictor j, the VIF is defined as where Rj2 is the coefficient of determination when predictor j is regressed against all other predictors. 13-5 Chapter 13 Multicollinearity Variance Inflation Factor (VIF) Some possible situations are: 13-6 Chapter 13 Multicollinearity Rules of Thumb 13-7 • There is no limit on the magnitude of the VIF. • A VIF of 10 says that the other predictors “explain” 90% of the variation in predictor j. • A high VIF indicates that predictor j is strongly related to the other predictors. • However, a high is not necessarily indicative of instability in the least squares estimate. • A large VIF is a warning to consider whether predictor j really belongs to the model. Chapter 13 Multicollinearity Are Coefficients Stable? • Evidence of instability is when X1 and X2 have a high pairwise correlation with Y, yet one or both predictors have insignificant t statistics in the fitted multiple regression, and/or if X1 and X2 are positively correlated with Y, yet one has a negative slope in the multiple regression. • As a test, try dropping a collinear predictor from the regression and see what happens to the fitted coefficients in the re-estimated model. • If they don’t change much, then multicollinearity is not a concern. • If there are sharp changes in one or more of the remaining coefficients in the model, then multicollinearity may be causing instability. 13-8 Chapter 13 Multicollinearity 13.2 Three Important Assumptions 1. The errors are normally distributed. 2. The errors have constant variance (i.e., they are homoscedastic). 3. The errors are independent (i.e., they are nonautocorrelated). Note: Everything you learned about residual tests in Chapter 12 is still true, except that the residuals are now based on multiple predictors (k > 2). This may affect the degrees of freedom in some tests, but the concepts are basically the same. 12-9 Chapter 13 Tests of Assumptions Non-Normal Errors • Non-normality of errors is a mild violation since the parameter estimates and their variances remain unbiased and consistent. • Confidence intervals for the parameters may be untrustworthy because the normality assumption is used to justify using Student’s t. Tests for Non-Normal Errors • • 12-10 Quick test: Make a histogram of residuals. Is it bell-shaped? Outliers? For a more precise test, use the normal probability plot. If H0 is true, the residual plot should be linear. H0: Errors are normally distributed H1: Errors are not normally distributed Chapter 13 Residual Tests Example: Normality Test (n = 50, k = 7) MegaStat’s normal probability plot is somewhat linear, but has some weird values at either end. Possible nonnormal residuals? But a histogram of residuals from Minitab's Stat > Graphical Summary is arguably normal in shape. From its p-value (.24) we would not reject normality. Note that the mean of the residuals is zero, as it must be. 12-11 Chapter 13 Residual Tests Heteroscedastic Errors (Nonconstant Variance) • Ideally, the error magnitude is constant (i.e., homoscedastic errors). Heteroscedastic errors increase or decrease with X. • In multiple regression, we have several X’s, so for simplicity we often just plot the n residuals against the fitted Y values. • In the most common form of heteroscedasticity, the variances of the estimators may be understated, the t-statistics overstated, and confidence intervals artificially narrow. Tests for Heteroscedasticity • 12-12 Plot the residuals against Yfitted or against each of the k predictors. Ideally, there is no pattern in the residuals moving from left to right. Chapter 13 Residual Tests Example: Regression with 7 Predictors • 12-13 Plots of residuals against Yfitted (shown below in blue) or against each of the k predictors (shown below in pink) show evidence of heteroscedasticity. Note that one predictor was binary (X = 0 or 1). Chapter 13 Residual Tests Autocorrelated Errors • Autocorrelation is a pattern of non-independent errors. • It is of more concern in time series data because the natural order of the data is meaningful (whereas in cross-sectional data the order of observatiohs if often alphabetical or randomized). • In a first-order autocorrelation, et is correlated with et 1. • The estimated variances of the OLS estimators are biased, resulting in confidence intervals that are too narrow, overstating the model’s fit. 12-14 Chapter 13 Residual Tests Runs Test for Autocorrelation • Look at a plot of residuals over time (or by observation). Count the number of sign reversals (i.e., how often does the residual cross the zero centerline?). • If the pattern is random, the number of sign changes should be near n/2. • Fewer than n/2 would suggest positive autocorrelation. • More than n/2 would suggest negative autocorrelation. Durbin-Watson (DW) Test • Tests for autocorrelation under the hypotheses H0: Errors are non-autocorrelated H1: Errors are autocorrelated • The DW statistic will range from 0 to 4. DW < 2 suggests positive autocorrelation DW = 2 suggests no autocorrelation (ideal) DW > 2 suggests negative autocorrelation 12-15 Chapter 13 Residual Tests Example: Normality Test (n = 50, k = 7) • Here is MegaStat’s plot of residuals by observation (n = 50). Count the number of sign reversals (i.e., how often does the residual cross the zero centerline). If the pattern is random, the number of sign changes should be near n/2 and the Durbin-Watson statistic should be near 2. DW is near 2 so there is not much evidence of autocorrelation. 28 sign changes (close to n/2 = 50/2 = 25) so there is not much evidence of autocorrelation. 12-16 Chapter 13 Residual Tests Example: Excel’s Tests of Assumptions Excel’s Data Analysis > Regression does residual plots (test for heteroscedasticity) and gives the DW test statistic. Excel’s standardized residuals are done in a strange way, but usually they are not misleading. Warning: Excel offers normal probability plots for residuals, but they are done incorrectly. 12-17 Chapter 13 Residual Tests Chapter 13 Residual Tests Example: MegaStat’s Tests of Assumptions MegaStat will do all three tests (if you check the boxes). Its runs plot (residuals by observation) is a visual test for autocorrelation, which Excel does not offer. 12-18 • Outliers? (omit only if clearly errors) • Missing Predictors? (usually you can’t tell) • Ill-Conditioned Data (adjust decimals or take logs) • Significance in Large Samples? (if n is huge, any regression will be significant) • Model Specification Errors? (may show up in residual patterns) • Missing Data? (we may have to live without it) • Binary Response? (if Y = 0,1 we use logistic regression) • Stepwise and Best Subsets Regression (MegaStat does these) 13-19 13-19 ML 13.3 Chapter 13 Other Regression Topics Tests for Nonlinearity • • • Sometimes the effect of a predictor is nonlinear. A simple example would be estimating the volume of lumber to be obtained from a tree. To test for suspected nonlinearity of any predictor variable, we can include its square in the regression. Tests for Interaction • We can test for interaction between two predictors by including their product in the regression. Model with x1x2 interaction term. 13-20 Chapter 13 Tests for Nonlinearity and Interaction Tests for Nonlinearity 13-21 Chapter 13 Tests for Nonlinearity and Interaction Example: MegaStat 13-22 Caution: This is basically a data-mining tool that looks only at fit (not at causal logic). Use it only as a check. Chapter 13 Stepwise Regression 13-23 Chapter 13 Chapter Summary 13-24 Chapter 13 Chapter Summary