Report

Z X T T W Y U Y using instrumental variables in education research SREE workshop march 2010 sean f reardon outline a little background on the potential outcomes framework what is an instrumental variable? and what’s it good for? assumptions needed to instrumental variables practical methods of estimating IV models sources of bias in IV models additional topics © 2010 by sean f. reardon. all rights reserved. potential outcomes framework a stylized example what is the effect of receiving tutoring in math on student math achievement? some made-up data for illustration: Observed Student Treatment and Achievement Data ID 1 2 3 4 5 6 Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring © 2010 by sean f. reardon. all rights reserved. Test Score 55 60 65 60 72 63 Observed Student Treatment and Achievement Data ID 1 2 3 4 5 6 Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring Test Score 55 60 65 60 72 63 Observed and Unobserved Potential Achievement Data Student ID 1 2 3 Untutored Average 4 5 6 Tutored Average Overall Average Treatment Condition no tutoring no tutoring no tutoring Score if not Tutored 55 60 65 60 tutoring tutoring tutoring © 2010 by sean f. reardon. all rights reserved. Score if Tutored 60 72 63 65 Observed Score 55 60 65 60 60 72 63 65 62.5 Tutoring Effect Definition of an “effect” The effect, , [on some outcome Y] [for some unit i] [of some treatment condition t relative to some other condition c] is defined as the difference between the value of Y that would be observed if unit i were exposed to treatment t and the value of Y that would be observed if unit i were exposed to treatment c. More formally, we define the effect of t relative to c on Y for unit i as: We define the average effect of t relative to c in a population P as: © 2010 by sean f. reardon. all rights reserved. The “Fundamental Problem of Causal Inference” (Holland, 1986) Although both and are defined in principle, it is impossible to observe both of them for the same unit (because any given unit can be exposed to only one of t or c). Thus, the causal effect cannot be observed. The problem of causal inference is thus a problem of missing data. The outcome Yi under its “counterfactual” condition is never observed. How can we construct unbiased estimates of the average potential outcomes and under the counterfactual conditions? © 2010 by sean f. reardon. all rights reserved. Observed Student Treatment and Achievement Data ID 1 2 3 4 5 6 Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring Test Score 55 60 65 60 72 63 Observed and Possible Unobserved Potential Achievement Data Student ID 1 2 3 Untutored Average 4 5 6 Tutored Average Overall Average Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring Score if not Tutored 55 60 65 60 55 60 65 60 60 © 2010 by sean f. reardon. all rights reserved. Score if Tutored 60 72 63 65 60 72 63 65 65 Observed Score 55 60 65 60 60 72 63 65 62.5 Tutoring Effect +5 +12 -2 +5 +5 +12 -2 +5 +5 Observed Student Treatment and Achievement Data ID 1 2 3 4 5 6 Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring Test Score 55 60 65 60 72 63 Observed and Possible Unobserved Potential Achievement Data Student ID 1 2 3 Untutored Average 4 5 6 Tutored Average Overall Average Treatment Condition no tutoring no tutoring no tutoring tutoring tutoring tutoring Score if not Tutored 55 60 65 60 55 70 70 65 62.5 © 2010 by sean f. reardon. all rights reserved. Score if Tutored 60 55 65 60 60 72 63 65 62.5 Observed Score 55 60 65 60 60 72 63 65 62.5 Tutoring Effect +5 -5 0 0 +5 +2 -7 0 0 What if we can’t conduct an RCT? If we can randomize students to receive either tutoring or no tutoring, and ensure that every student complies with his or her assigned treatment status, the randomization will allow us to estimate the effect of tutoring very easily. but what if students don’t comply with their treatment assignment? some assigned to tutoring don’t go to tutoring some assigned to no tutoring get tutored anyway this means tutoring is no longer randomly assigned – at least some of the variation in treatment status is potentially endogenous so a comparison of those assigned to tutoring and no tutoring won’t give us an estimate of the effect of tutoring (but only the effect of being assigned to tutoring) this is one case where instrumental variables are useful instrumental variables models What is an instrumental variable? an instrumental variable is an exogenous factor that causes some of the variation in treatment status (though need not be all) we use it to identify the portion of variation in treatment that is exogenous and then only rely on that exogenous variation to estimate the effect of treatment © 2010 by sean f. reardon. all rights reserved. A general structural model Z X T T W Y U Y © 2010 by sean f. reardon. all rights reserved. T: treatment status Y: outcome measure X: observed confounders U: unobserved confounders W: observed ignorable causes of Y Y: unobserved ignorable causes of Y T: unobserved ignorable causes of T Z: instrument (observed ignorable cause of T) Relating treatments and outcomes T Y we would like to estimate the effect of T on Y this involves seeing how T and Y are related but to infer a causal relationship from the covariance of T and Y, we need to understand the source of variation in T © 2010 by sean f. reardon. all rights reserved. why do some people get different types/degrees of the treatment? Relating treatments and outcomes Z variation in T may be caused by factors unrelated to the outcome Y T Y T © 2010 by sean f. reardon. all rights reserved. these may be observed (Z) or unobserved (T) if the only variation in Z comes from factors unrelated to Y, then T is as good as randomly assigned, so getting a causal estimate is easy Relating treatments and outcomes X variation in T may be caused, in part, by observed factors that are related to the outcome Y T Y T as long as there is some variation in T that is caused by some (not necessarily observable) ignorable cause (Z or T), we can still easily get an estimate of the effect of T © 2010 by sean f. reardon. all rights reserved. observed confounders (X) statistically control for X (compute relationship between T and Y, conditional on X) Relating treatments and outcomes X T variation in T may be caused, in part, by observed and unobserved factors that are related to the outcome Y Y T U here, we cannot get an unbiased estimate of the effect of T © 2010 by sean f. reardon. all rights reserved. observed confounders (X) unobserved confounders (U) reverse causality (Y affects T) statistical control can’t adjust for U the ignorable cause (T) is not observed Relating treatments and outcomes Z X T Y T U © 2010 by sean f. reardon. all rights reserved. if we cannot observe all the confounders (or if Y affects T), then we need some observed factor that affects T but does not otherwise affect Y this (Z) is called an instrument (or instrumental variable). because the part of the variation in T that is induced is ignorable (as good as random), we can use this part of the variation in T to identify the effect of T on Y Tutoring example, revisited the observed data is not sufficient to estimate the average effect of tutoring what if we can’t do an experiment, or if we do an experiment and not everyone complies? © 2010 by sean f. reardon. all rights reserved. tutoring voucher as an instrument randomly assign eligible students to receive a either voucher allowing them to receive free tutoring (Z=1) or no voucher (Z=0). observe whether students attend tutoring (T=1) or not (T=0). note: this choice is not random—students may choose tutoring or not, regardless of voucher status (Ti≠Zi). observe later achievement (Y) we want to estimate the effect of T (tutoring vs no tutoring) on Y (achievement). © 2010 by sean f. reardon. all rights reserved. Four subpopulations (angrist, imbens, & rubin, 1996) compliers those who would comply with treatment assignment (those for whom Ti=Zi) non-compliers always-takers never-takers those who would always receive the treatment, regardless of assignment (those for whom Ti=1) those who would never receive the treatment, regardless of assignment (those for whom Ti=0) defiers those who would always do the opposite of treatment assignment (those for whom Ti=1-Zi) © 2010 by sean f. reardon. all rights reserved. Observed Outcomes N=100, 50% receive vouchers, but not all comply with assignment (only 60% comply): Offered Voucher No Yes Total Tutored No Yes 45 5 15 35 60 40 © 2010 by sean f. reardon. all rights reserved. Proportion Tutored .10 .70 .40 Observed Outcomes N=100, 50% receive vouchers, but not all comply with assignment (only 60% comply): Offered Voucher No Yes Total Tutored No Yes 45 5 15 35 60 40 © 2010 by sean f. reardon. all rights reserved. Proportion Tutored might be compliers or never-takers .10 .70 .40 Observed Outcomes N=100, 50% receive vouchers, but not all comply with assignment (only 60% comply): Offered Voucher No Yes Total Tutored No Yes 45 5 15 35 60 40 © 2010 by sean f. reardon. all rights reserved. Proportion Tutored might be .10 defiers or never-takers .70 .40 Observed Outcomes N=100, 50% receive vouchers, but not all comply with assignment (only 60% comply): Offered Voucher No Yes Total Tutored No Yes 45 5 15 35 60 40 © 2010 by sean f. reardon. all rights reserved. Proportion Tutored might be defiers or always-takers .10 .70 .40 Observed Outcomes N=100, 50% receive vouchers, but not all comply with assignment (only 60% comply): Offered Voucher No Yes Total Tutored No Yes 45 5 15 35 60 40 © 2010 by sean f. reardon. all rights reserved. Proportion Tutored might be .10 compliers or always-takers .70 .40 estimating the proportion of compliers assume there are no defiers then everyone with Z=1, T=0 is a never-taker (15 of 50 (30%) with Z=1 in our example) there should be the same proportion (30%) of never-takers among those with Z=0, because Z is random the same logic implies there are 10% of the population who are always-takers thus, 60% (100% - 30% - 10%) are compliers © 2010 by sean f. reardon. all rights reserved. Estimating the proportion of compliers we can also estimate this by regressing the treatment variable on the instrument tutor = G0 + G1*voucher + e tutor = .10 + 0.60*voucher Thus, the average effect of being assigned a voucher on tutoring status is +0.60, meaning that the average student’s probability of receiving tutoring increases by 0.60 if assigned a voucher (which means that 60% of the students comply with the voucher assignment). © 2010 by sean f. reardon. all rights reserved. Observed Outcomes Estimated effect of the voucher offer on test scores = 56.6 – 50.5 = +6.1 Offered Tutored Voucher No Yes Total No 48.3 70.0 50.5 Yes 44.9 61.6 56.6 Total 47.5 62.6 53.5 © 2010 by sean f. reardon. all rights reserved. Observed Outcomes Estimated effect of the voucher offer on test scores = 56.6 – 50.5 = +6.1 Offered Tutored here we’re Voucher No Yes Total assuming no defiers No 48.3 70.0 50.5 (later we will see why Yes 44.9 61.6 56.6 this is necessary) Total 47.5 62.6 53.5 average outcome among untutored compliers and never-takers © 2010 by sean f. reardon. all rights reserved. Observed Outcomes Estimated effect of the voucher offer on test scores = 56.6 – 50.5 = +6.1 Offered Tutored here we’re Voucher No Yes Total assuming no defiers No 48.3 70.0 50.5 (later we will see why Yes 44.9 61.6 56.6 this is necessary) Total 47.5 62.6 53.5 average outcome among untutored compliers and never-takers © 2010 by sean f. reardon. all rights reserved. average outcome among tutored compliers and always-takers OLS estimates OLS yields: test = 47.5 + 15.1*(tutored) the estimated effect of tutoring is +15.1 points but we should worry about whether this is biased, because some students chose whether to get tutoring or not. the tutored group includes compliers and alwaystakers; the control group includes compliers and never-takers; so they are not equivalent groups © 2010 by sean f. reardon. all rights reserved. The Wald IV estimator if we are willing to assume that the voucher offer had no effect on the outcome of the noncompliers (because it did not alter their treatment status and does not affect their outcome through any other way), then we can estimate the effect of tutoring like this: The average effect of the voucher in the population is estimated to be +6.1 but only 60% of students’ decisions about whether to get tutoring were affected by the voucher offer (only 60% of sample are compliers) © 2010 by sean f. reardon. all rights reserved. Wald estimator average effect in population ( ) = average effect on compliers ( ) x proportion who are compliers ( ) + average effect on non-compliers ( ) x proportion who are non-compliers ( ) © 2010 by sean f. reardon. all rights reserved. Wald estimator this says that the average effect of the treatment among the compliers equals the average effect in the population divided by the proportion of the population who are compliers thus, the average effect among the compliers is = +6.1/.60 = +10.1 © 2010 by sean f. reardon. all rights reserved. What have we learned? An instrumental variable allows us to estimate the average effect of the treatment among those whose treatment status is affected by the instrument (“compliers”) called the “local average treatment effect” (LATE) note that we can’t identify who the compliers are We can’t estimate the average treatment effect in the population, because we can’t estimate the effect among non-compliers because the instrument doesn’t affect their treatment status, there is no exogenous variation in their treatment status that we can use. © 2010 by sean f. reardon. all rights reserved. What assumptions have we made? the instrument only affects the outcome through its impact on the treatment (this is called the exclusion restriction) the instrument is ignorably (randomly) assigned this allows us to estimate the effect of the instrument on the outcome and on the treatment the instrument affects the treatment for at least some people otherwise there are no compliers there are no defiers © 2010 by sean f. reardon. all rights reserved. more general IV models what if treatment is not binary? above we assumed the treatment (tutoring) was binary but not all treatments are binary we could offer vouchers of different amounts students could receive different amounts of tutoring as a result, compliance may take on many values for some students, the amount of tutoring received may be strongly affected by the instrument; for others, it may be weakly affected or not at all affected. © 2010 by sean f. reardon. all rights reserved. a more general model of the IV estimator for a given individual i, is the effect of Z on Y this effect may vary across individuals we would like to estimate the average effect, © 2010 by sean f. reardon. all rights reserved. Zi i Yi 1. exclusion restriction if the only way that Z affects Y is through its effect on T, then we have . or, put differently, Zi γi Ti i Yi the assumption that the only way that Z affects Y is through its effect on T is called the exclusion restriction. © 2010 by sean f. reardon. all rights reserved. 2. zero compliance-effect covariance we can write the average effect of Z on Y as if we assume the assumption that is called the zero compliance-effect covariance assumption. © 2010 by sean f. reardon. all rights reserved. , then we have 3. instrument relevance as long as the assumption that is sometimes called the instrument relevance assumption; or sometimes just referred to as the assumption that the instrument affects the treatment. if is small (close to zero), we say that the instrument is a weak instrument. , we can rewrite the above as © 2010 by sean f. reardon. all rights reserved. 4. the instrument is ignorably assigned if the above three assumptions are met, we have if Z is ignorably assigned, then we can easily estimate both (the average effect of Z on Y) and (the average effect of Z on T). the assumption of ignorable assignment thus makes estimation of the effect of T on Y possible. © 2010 by sean f. reardon. all rights reserved. what do these assumptions mean? exclusion restriction: the offer of a tutoring voucher does not affect students’ achievement except by affecting the amount of tutoring they receive zero compliance-effect covariance: there is no correlation between how strongly a voucher offer affects the amount of tutoring a student gets and how effective tutoring is for that student © 2010 by sean f. reardon. all rights reserved. what do these assumptions mean? instrument relevance: the offer of a voucher has some effect, on average, on the amount of tutoring students receive (at least one student is affected by the offer). ignorable assignment of the instrument: the voucher offer is randomly assigned (this would be violated, for example, if the principal gave vouchers to students she deemed most in need of tutoring). © 2010 by sean f. reardon. all rights reserved. some examples NYC voucher experiment (howell et al, 2002; krueger & zhu, 2004) Effect of schooling on wages, using quarter of birth as instrument (angrist & kreuger, 1991). Effect of teacher absence on student achievement, using snowfall as instrument (miller, murnane & willet, 2007) Effects of segregation on educational attainment and wages, using railroads as an instrument (ananat 2007) © 2010 by sean f. reardon. all rights reserved. estimating IV models estimating IV models in practice in practice, we don’t usually compute the effect of Z on Y and Z on T and divide them because we made need more complex models (if we want to include other covariates in the model, for example) because we need to compute standard errors most common methods of estimating IV models is with two-stage least squares (TSLS or 2SLS). © 2010 by sean f. reardon. all rights reserved. Three relevant equations 1: is the person-specific effect of Z on Y. 2: is the person-specific effect of Z on T. but the equation we really are interested in is 3: is the person-specific effect of T on Y. © 2010 by sean f. reardon. all rights reserved. Three relevant equations 1: is the person-specific effect of Z on Y. the “reduced form” equation 2: is the person-specific effect of Z on T. but the equation we really are interested in is 3: is the person-specific effect of T on Y. © 2010 by sean f. reardon. all rights reserved. Three relevant equations 1: is the person-specific effect of Z on Y. the “reduced form” equation the “first stage” equation 2: is the person-specific effect of Z on T. but the equation we really are interested in is 3: is the person-specific effect of T on Y. © 2010 by sean f. reardon. all rights reserved. Three relevant equations 1: is the person-specific effect of Z on Y. the “reduced form” equation the “first stage” equation 2: is the person-specific effect of Z on T. but the equation we really are interested in is the “second stage” 3: equation is the person-specific effect of T on Y. © 2010 by sean f. reardon. all rights reserved. two-stage least squares fit the first-stage equation (estimate the effect of Z on T); compute fitted values: fit the second-stage equation, using predicted values of T in place of observed values of T: © 2010 by sean f. reardon. all rights reserved. two-stage least squares fit the first-stage equation (estimate the effect of Z on T); compute fitted values: fit the second-stage equation, using predicted values of T in place of observed values of T: © 2010 by sean f. reardon. all rights reserved. two-stage least squares fit the first-stage equation (estimate the effect of Z on T); compute fitted values: fit the second-stage equation, using predicted values of T in place of observed values of T: © 2010 by sean f. reardon. all rights reserved. two-stage least squares because the predicted values of T from the firststage equation include only the variation in T that is caused by the instrument, the estimated coefficient from the second-stage equation will be unbiased (as long as the 4 IV assumptions are met). if you do this by hand, you’ll get the wrong standard errors; statistical software usually has built-in routines (e.g., -ivregress- command in Stata) to compute correct standard errors. © 2010 by sean f. reardon. all rights reserved. Effects of attending charter school we can’t randomize students to charter or traditional public schools Abdulkadiroglu, et al (2009) examine students who apply to oversubscribed charter schools, whose admission is determined by lottery (randomization) instrument is winning the lottery treatment is # of years in a charter school © 2010 by sean f. reardon. all rights reserved. example: effect of charter schooling first stage (compliance) © 2010 by sean f. reardon. all rights reserved. reduced form 2sls (effect of winning (effect of a lottery on ach.) year in charter) are the IV assumptions valid in this study? exclusion restriction? zero compliance-effect covariance? instrument relevance? ignorable assignment? © 2010 by sean f. reardon. all rights reserved. sources of bias in IV models sources of bias in IV failure of exclusion restriction assumption failure of ignorability assumption failure of zero compliance-effect covariance assumption finite sample bias weak instruments cause 3 problems: exacerbate bias due to failure of assumptions (exclusion restriction, ignorability, zero covariance) exacerbate finite sample bias lead to incorrect estimation of standard errors when using twostage least squares © 2010 by sean f. reardon. all rights reserved. failure of the exclusion restriction recall that the exclusion restriction says that the only way that Z affects Y is through its effect on T. as a result, we can write © 2010 by sean f. reardon. all rights reserved. Zi γi Ti i Yi failure of the exclusion restriction if the exclusion restriction is violated, then there is some other path through which Z affects Y as a result, we can write © 2010 by sean f. reardon. all rights reserved. Zi i γi Ti i Yi failure of the zero covariance assumption averaging the above in the population now, dividing through by , we get so the IV estimator (the ratio of the average effect of Z on Y to the average effect of Z on T) will be biased if is small, the biases will be larger © 2010 by sean f. reardon. all rights reserved. failure of the zero covariance assumption averaging the above in the population now, dividing through by , we get so the IV estimator (the ratio of the average effect of Z on Y to the average effect of Z on T) will be biased if is small, the biases will be larger © 2010 by sean f. reardon. all rights reserved. bias due to failure of the exclusion restriction bias due to failure of the zero compliance-effect covariation assumption failure of the zero covariance assumption if all the assumptions except the zero compliance-effect covariance assumption are met, we have so the IV model will estimate the complianceweighted average treatment effect (CWATE). if T is binary and there are no defiers, this will be the same as the average effect among the compliers (LATE), because non-compliers will get 0 weight. © 2010 by sean f. reardon. all rights reserved. failure of the ignorability assumption if the instrument is not ignorably assigned, then we cannot obtain unbiased estimates of the effect of Z on Y or of the effect of Z on T. Thus, the ratio of the two may be biased. © 2010 by sean f. reardon. all rights reserved. weak instruments weak instruments do not, strictly-speaking, violate any of the IV assumptions, but they do exacerbate the bias from other assumptions rule of thumb: an instrument is weak if the Fstatistic on the instrument(s) from the first stage equation is <10. © 2010 by sean f. reardon. all rights reserved. weak instruments and bias the IV estimator weak instruments cause 3 problems with IV estimator: exacerbate bias due to failure of the exclusion restriction, ignorability, and monotonicity exacerbate finite sample bias lead to incorrect estimation of standard errors when using two-stage least squares finite sample bias even if the 4 IV assumptions are met, IV estimation is biased unless using an infinite sample most pronounced with weak instruments and small samples © 2010 by sean f. reardon. all rights reserved. additional uses mediation models suppose we randomly assign a treatment (e.g., teacher professional development) that we think will affect student learning by affecting instructional practice we can treat the PD as an instrument, and the mediator (instructional practice) as the ‘treatment’ and use IV to estimate the effect of instructional practice (which can’t be randomized) on learning but worry about exclusion restriction (are there other ways that the PD could affect learning?) © 2010 by sean f. reardon. all rights reserved. multiple mediator models suppose we have a randomize students to 3 treatment conditions. two first stage equations: second stage equation: © 2010 by sean f. reardon. all rights reserved. IV to correct for measurement error suppose we want to estimate the effect of cognitive skill on wages: if cognitive skill is measured with error by ACH, OLS will give a biased estimate of . if we have a second test of skills, we can use one test as an instrument for the second test, and then use the predicted value of the second test in the wage equation. called “errors-in-variables” (EIV) model. © 2010 by sean f. reardon. all rights reserved.