Report

Week 12 November 17-21 Four Mini-Lectures QMM 510 Fall 2014 ML 12.1 Chapter Contents 13.1 Multiple Regression 13.2 Assessing Overall Fit 13.3 Predictor Significance 13.4 Confidence Intervals for Y 13.5 Categorical Predictors 13.6 Tests for Nonlinearity and Interaction 13.7 Multicollinearity 13.8 Violations of Assumptions 13.9 Other Regression Topics 13-2 Much of this is like Chapter 12, except that we have more than one predictor. Chapter 13 Multiple Regression Chapter 13 Multiple Regression Simple or Multivariate? • Multiple regression is an extension of simple regression to include more than one independent variable. • Limitations of simple regression: • often simplistic • biased estimates if relevant predictors are omitted • lack of fit does not show that X is unrelated to Y if the true model is multivariate 13-3 Chapter 13 Multiple Regression Visualizing a Multiple Regression 13-4 Chapter 13 Multiple Regression Regression Terminology • Y is the response variable and is assumed to be related to the k predictors (X1, X2, … Xk) by a linear equation called the population regression model: Use Greek letters for population parameters • The estimated (fitted) regression equation is: Use Roman letters for sample estimates 13-5 Chapter 13 Multiple Regression Fitted Regression: Simple versus Multivariate If we have more than two predictors, there is no way to visualize it … 13-6 Chapter 13 Multiple Regression Data Format n observed values of the response variable Y and its proposed predictors X1, X2, …, Xk are presented in the form of an n x k matrix. 13-7 Chapter 13 Multiple Regression Common Misconceptions about Fit • A common mistake is to assume that the model with the best fit is preferred. • Sometimes a model with a low R2 may give useful predictions, while a model with a high R2 may conceal problems. • Thoroughly analyze the results before choosing the model. 13-8 Chapter 13 Multiple Regression Four Criteria for Regression Assessment • Logic - Is there an a priori reason to expect a causal relationship between the predictors and the response variable? • Fit - Does the overall regression show a significant relationship between the predictors and the response variable? • Parsimony - Does each predictor contribute significantly to the explanation? Are some predictors not worth the trouble? • Stability - Are the predictors related to one another so strongly that the regression estimates become erratic? 13-9 Chapter 13 Assessing Overall Fit F Test for Significance • • For a regression with k predictors, the hypotheses to be tested are H0: All the true coefficients are zero H1: At least one of the coefficients is nonzero In other words, H0: b1 = b2 = … = bk= 0 H1: At least one of the coefficients is nonzero 13-10 Chapter 13 Assessing Overall Fit F Test for Significance The ANOVA calculations for a k-predictor model resemble those for a simple regression, except for degrees of freedom: 13-11 Chapter 13 Assessing Overall Fit Coefficient of Determination (R2) • R2, the coefficient of determination, is a common measure of overall fit. • It can be calculated in one of two ways (always done by computer). • For example, for the home price data, 13-12 Chapter 13 Assessing Overall Fit Adjusted R2 • It is generally possible to raise the coefficient of determination R2 by including additional predictors. • The adjusted coefficient of determination is done to penalize the inclusion of useless predictors. • For n observations and k predictors: 13-13 Chapter 13 Assessing Overall Fit How Many Predictors? • Limit the number of predictors based on the sample size. • A large sample size permits many predictors. • When n/k is small, the R2 no longer gives a reliable indication of fit. • Suggested rules are: Evan’s Rule (conservative): n/k 0 (at least 10 observations per predictor) Doane’s Rule (relaxed): n/k 5 (at least 5 observations predictor) These are just guidelines – use your judgment. 13-14 Chapter 13 Predictor Significance • Test each fitted coefficient to see whether it is significantly different from zero. • The hypothesis tests for the coefficient of predictor Xj are • If we cannot reject the hypothesis that a coefficient is zero, then the corresponding predictor does not contribute to the prediction of Y. 13-15 Chapter 13 Predictor Significance Test Statistic • Excel reports the test statistic for the coefficient of predictor Xj : • Find the critical value tα for chosen level of significance α from Appendix D or from Excel using =T.INV.2T(α,df) 2 tailed test. • To reject H0 we compare tcalc to tα for the different hypotheses (or reject if p-value α). • The 95% confidence interval for coefficient bj is 13-16 Chapter 13 Confidence Intervals for Y Standard Error • The standard error of the regression (se) is another important measure of fit. Except for d.f. the formula for se resembles se for simple regression. • For n observations and k predictors • If all predictions were perfect (SSE = 0) then se = 0. 13-17 Chapter 13 Confidence Intervals for Y Approximate Confidence and Prediction Intervals for Y • Approximate 95% confidence interval for conditional mean of Y: • Approximate 95% prediction interval for individual Y value: 13-18 Chapter 13 Confidence Intervals for Y Quick 95 Percent Confidence and Prediction Interval for Y • The t-values for 95% confidence are typically near 2 (as long as n is not too small). • Very quick prediction and confidence intervals for Y interval without using a t table are: 13-19 ML 12.2 Standardized Residuals • Use Excel, MINITAB, MegaStat or other software to compute standardized residuals. • If the absolute value of any standardized residual is at least 2, then it is classified as unusual (as in simple regression). Leverage and Influence • A high leverage statistic indicates unusual X values in one or more predictors. • Such observations are influential because they are near the edge(s) of the fitted regression plane. • Leverage for observation i is denoted hi (computed by MegaStat) 12-20 Chapter 13 Unusual Observations Leverage • For a regression model with k predictors, an observation whose leverage exceeds 2(k+1)/n is unusual. • In Chapter 12, the leverage rule was 4/n. With k = 1 predictor, we get 2(k+1)/n = 2(1+1)/n = 4/n. • So this leverage criterion applies to simple regression as a special case. 12-21 Chapter 13 Unusual Observations Example: Heart Death Rate in 50 States standard error se = 27.422 n = 50 states, k = 3 predictors 4 states (FL, HI, OK, WV) have unusual residuals (> 2 se) highlighted by MegaStat high leverage criterion is 2(k+1)/n = 2(3+1)/50 = 0.160 Note: Only unusual observations are shown (there were n = 50 observations) 12-22 MegaStat highlights the high leverage observations (> .160) Chapter 13 Unusual Observations Chapter 13 Categorical Predictors ML 12.3 What Is a Binary or Categorical Predictor? • A binary predictor has two values (usually 0 and 1) to denote the presence or absence of a condition. • For example, for n graduates from an MBA program: Employed = 1 Unemployed = 0 • These variables are also called dummy , dichotomous, or indicator variables. • For easy understandability, name the binary variable the characteristic that is equivalent to the value of 1. 13-23 Chapter 13 Categorical Predictors Effects of a Binary Predictor • A binary predictor is sometimes called a shift variable because it shifts the regression plane up or down. • Suppose X1 is a binary predictor that can take on only the values of 0 or 1. • Its contribution to the regression is either b1 or nothing, resulting in an intercept of either b0 (when X1 = 0) or b0 + b1 (when X1 = 1). • The slope does not change: only the intercept is shifted. For example, 13-24 Testing a Binary for Significance • In multiple regression, binary predictors require no special treatment. They are tested as any other predictor using a t test. More Than One Binary • More than one binary occurs when the number of categories to be coded exceeds two. • For example, for the variable GPA by class level, each category is a binary variable: Freshman = 1 if a freshman, 0 otherwise Sophomore = 1 if a sophomore, 0 otherwise Junior = 1 if a junior, 0 otherwise Senior = 1 if a senior, 0 otherwise Masters = 1 if a master’s candidate, 0 otherwise Doctoral = 1 if a PhD candidate, 0 otherwise 13-25 Chapter 13 Categorical Predictors What if I Forget to Exclude One Binary? • Including all binaries for all categories may introduce a serious problem of collinearity for the regression estimation. Collinearity occurs when there are redundant independent variables. • When the value of one independent variable can be determined from the values of other independent variables, one column in the X data matrix will be a perfect linear combination of the other column(s). • The least squares estimation would fail because the data matrix would be singular (i.e., would have no inverse). 13-26 Chapter 13 Categorical Predictors • Outliers? (omit only if clearly errors) • Missing Predictors? (usually you can’t tell) • Ill-Conditioned Data (adjust decimals or take logs) • Significance in Large Samples? (if n is huge, almost any regression will be significant) • Model Specification Errors? (may show up in residual patterns) • Missing Data? (we may have to live without it) • Binary Response? (if Y = 0,1 we use logistic regression) • Stepwise and Best Subsets Regression (MegaStat does these) 13-27 13-27 Chapter 13 Other Regression Problems