Report

LECTURE 3 Introduction to Linear Regression and Correlation Analysis 1 Simple Linear Regression 2 Regression Analysis 3 Regression Model Validity Goals After this, you should be able to: Interpret the simple linear regression equation for a set of data Use descriptive statistics to describe the relationship between X and Y Determine significant whether a regression model is Goals (continued) After this, you should be able to: Interpret confidence intervals for the regression coefficients Interpret confidence intervals for a predicted value of Y Check whether regression assumptions are satisfied Check to see if the data contains unusual values Introduction to Regression Analysis Regression analysis is used to: Predict the value of a dependent variable based on the value of at least one independent variable Explain the impact of changes in an independent variable on the dependent variable Dependent variable: the variable we wish to explain Independent variable: the variable used to explain the dependent variable Simple Linear Regression Model Only one independent variable, x Relationship between x and y is described by a linear function Changes in y are assumed to be caused by changes in x Types of Regression Models Positive Linear Relationship Negative Linear Relationship Relationship NOT Linear No Relationship Population Linear Regression The population regression model: Population y intercept Dependent Variable Population Slope Coefficient Independent Variable y β0 β1x ε Linear component Random Error term, or residual Random Error component Linear Regression Assumptions The underlying relationship between the x variable and the y variable is linear The distribution of the errors has constant variability Error values are normally distributed Error values are independent (over time) Population Linear Regression y y β0 β1x ε Observed Value of y for xi εi Predicted Value of y for xi Slope = β1 Random Error for this x value Intercept = β0 xi x Estimated Regression Model The sample regression line provides an estimate of the population regression line Estimated (or predicted) y value Estimate of the regression intercept Estimate of the regression slope yˆ i b0 b1x Independent variable Interpretation of the Slope and the Intercept b0 is the estimated average value of y when the value of x is zero b1 is the estimated change in the average value of y as a result of a one-unit change in x Finding the Least Squares Equation The coefficients b0 and b1 will be found using computer software, such as Excel’s data analysis add-in or MegaStat Other regression measures will also be computed as part of computerbased regression analysis Simple Linear Regression Example A real estate agent wishes to examine the relationship between the selling price of a home and its size (measured in square feet) A random sample of 10 houses is selected Dependent variable (y) = house price in $1000 Independent variable (x) = square feet Sample Data for House Price Model House Price in $1000s (y) Square Feet (x) 245 1400 312 1600 279 1700 308 1875 199 1100 219 1550 405 2350 324 2450 319 1425 255 1700 Regression output from Excel – Data – Data Analysis or MegaStat – Correlation/ regression MegaStat – Correlation/ regression MegaStat Output The regression equation is: Predicted house price 98.24833 0.10977 (square feet) Regression Analysis r² 0.581 r 0.762 Std. Error 41.330 ANOVA table Source SS Regression 18,934.9348 Residual 13,665.5652 Total 32,600.5000 Regression output variables coefficients Intercept 98.2483 Square feet 0.1098 n 10 k1 Dep. Var. Price($000) df MS 1 18,934.9348 8 1,708.1957 9 F 11.08 std. error t (df=8) p-value 0.0330 3.329 .0104 p-value .0104 confidence interval 95% lower 95% upper 0.0337 0.1858 Graphical Presentation House price model: scatter plot and regression line 450 House Price ($1000s) Intercept = 98.248 400 350 Slope = 0.10977 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 Square Feet house price 98.24833 0.10977 (squarefeet) Interpretation of the Intercept, b0 house price 98.24833 0.10977 (squarefeet) b0 is the estimated average value of Y when the value of X is zero (if x = 0 is in the range of observed x values) Here, houses with 0 square feet do not occur, so b0 = 98.24833 just indicates the height of the line. Interpretation of the Slope Coefficient, b1 house price 98.24833 0.10977 (squarefeet) b1 measures the estimated change in Y as a result of a one-unit increase in X Here, b1 = .10977 tells us that the average value of a house increases by .10977($1000) = $109.77, on average, for each additional one square foot of size Least Squares Regression Properties The simple regression line always passes through the mean of the y variable and the mean of the x variable The least squares coefficients are unbiased estimates of β0 and β1 Coefficient of Determination, R2 The percentage of variability in Y that can be explained by variability in X. Note: In the single independent variable case, the coefficient of determination is R r 2 2 where: R2 = Coefficient of determination r = Simple correlation coefficient Examples of R2 Values y R2 = 1, correlation = -1 R2 = 1 y R2 x Perfect linear relationship between x and y: 100% of the variation in y is explained by variation in x x = 1, correlation = +1 Examples of Approximate R2 Values y 0 < R2 < 1, correlation is negative x Weaker linear relationship between x and y: Some but not all of the variation in y is explained by variation in x y 0 < R2 < 1, correlation is positive x Examples of Approximate R2 Values R2 = 0 y No linear relationship between x and y: R2 = 0 x The value of Y does not depend on x. (None of the variation in y is explained by variation in x) Excel Output Regression Analysis r² 0.581 r 0.762 Std. Error 41.330 58.08% of the variation in house prices is explained by variation in square feet The correlation of .762 shows a fairly strong direct relationship. The typical error in predicting Price is 41.33($000) = $41,330 Inference about the Slope: t Test t test for a population slope Null and alternative hypotheses Is there a linear relationship between x and y? H0: β1 = 0 Ha: β1 0 (no linear relationship) (linear relationship does exist) Obtain p-value from ANOVA or across from the slope coefficient (they are the same in simple regression) Inference about the Slope: t Test House Price in $1000s (y) Square Feet (x) 245 1400 312 1600 279 1700 308 1875 199 1100 219 1550 405 2350 324 2450 319 1425 255 1700 (continued) Estimated Regression Equation: house price 98.25 0.1098 (sq.ft.) The slope of this model is 0.1098 Does square footage of the house affect its sales price? Inferences about the Slope: t Test Example H0: β1 = 0 Ha: β1 0 P-value From Excel output: Coefficients Intercept Square Feet Standard Error t Stat P-value 98.24833 58.03348 1.69296 0.12892 0.10977 0.03297 3.32938 0.01039 Decision: Reject H0 Conclusion: We can be 98.96% confident that square feet is related to house price. Regression Analysis for Description Confidence Interval Estimate of the Slope: Coefficient Standard Excel Printout forsHouse Prices: Error t Stat P-value Lower 95% Intercept Square Feet Upper 95% 98.24833 58.03348 1.69296 0.12892 -35.57720 232.07386 0.10977 0.03297 3.32938 0.01039 0.03374 0.18580 We can be 95% confident that house prices increase by between $33.74 and $185.80 for a 1 square foot increase. Estimates of Expected y for Different Values of x y yp y The relationship describes how x impacts your estimate from y x xp x Interval Estimates for Different Values of x y Prediction Interval for an individual y, given xp The father from x the less accurate the prediction. y x xp x Example: House Prices House Price in $1000s (y) Square Feet (x) 245 1400 312 1600 279 1700 308 1875 199 1100 219 1550 405 2350 324 2450 319 1425 255 1700 Estimated Regression Equation: house price 98.25 0.1098 (sq.ft.) Predict the price for a house with 2000 square feet Example: House Prices (continued) Predict the price for a house with 2000 square feet: house price 98.25 0.1098 (sq.ft.) 98.25 0.1098(2000) 317.85 The predicted price for a house with 2000 square feet is 317.85($1,000s) = $317,850 Estimation of Individual Values: Example Prediction Interval Estimate for y|xp Find the 95% confidence interval for an individual house with 2,000 square feet Predicted Price Yi = 317.85 ($1,000s) = $317, 850 MegaStat will give both the predicted value as well as the lower and upper limits Predicted values for: Price($000) 95% Confidence Interval Square feet Predicted lower upper 2,000 317.784 280.664 354.903 95% Prediction Interval lower 215.503 upper 420.065 The prediction interval endpoints are from $215,503 to $420,065. We can be 95% confident that the price of a 2000 ft2 home will fall within those limits. Residual Analysis Purposes Check for linearity assumption Check for the constant variability assumption for all levels of predicted Y Check normal residuals assumption Check for independence over time Graphical Analysis of Residuals Can plot residuals vs. x and predicted Y Can create NPP of residuals to check for normality (or use Skewness/Kurtosis) Can check D-W statistic to confirm independence Residual Analysis for Linearity y y x x Not Linear residuals residuals x x Linear Residual Analysis for Constant Variance y y x Ŷ Non-constant variance residuals residuals x Ŷ Constant variance Residual Analysis for Normality Can create NPP of residuals to check for normality. If you see an approximate straight line residuals are acceptably normal. You can also use Skewness/Kurtosis. If both are within + 1 the residuals are acceptably normal Residual Analysis for Independence – Can check D-W statistic to confirm independence. If D-W statistic is greater than 1.3 the residuals are acceptably independent. Needed only if the data is collected over time. Checking Unusual Data Points Check for outliers from the predicted values (studentized and studentized deleted residuals do this; MegaStat highlights in blue) Check for outliers on the X-axis; they are indicated by large leverage values; more than twice as large as the average leverage. MegaStat highlights in blue. Check Cook’s Distance which measures the harmful influence of a data point on the equation by looking at residuals and leverage together. Cook’s D > 1 suggests potentially harmful data points and those points should be checked for data entry error. MegaStat highlights in blue based on F distribution values. Patterns of Outliers a). Outlier is extreme in both X and Y but not in pattern. The point is unlikely to alter regression line. b). Outlier is extreme in both X and Y as well as in the overall pattern. This point will strongly influence regression line c). Outlier is extreme for X nearly average for Y. The further it is away from the pattern the more it will change the regression. d). Outlier extreme in Y not in X. The further it is away from the pattern the more it will change the regression. e). Outlier extreme in pattern, but not in X or Y. Slope may not be changed much but intercept will be higher with this point included. Summary Introduced simple linear regression analysis Calculated the coefficients for the simple linear regression equation measures of strength (r, R2 and se) Summary (continued) Described inference about the slope Addressed prediction of individual values Discussed residual analysis to address assumptions of regression and correlation Discussed checks for unusual data points