Report

The estimation strategy of the National Household Survey (NHS) François Verret, Mike Bankier, Wesley Benjamin & Lisa Hayden Statistics Canada Presentation at the ITSEW 2011 June 21, 2011 Outline of the presentation 1. Introduction 2. Handling non-response error 3. Simulation set-up 4. Results 5. Limits of the study 6. Conclusion 7. Future work 2 2 1. Introduction 2006 Census: 20% long form, 80% short form 2011: • 100% Census mandatory short form • 30% sampled to voluntarily complete the NHS long form Objectives of the long form: get data to plan, deliver and support government programs directed at target populations 2011 common topics to both forms: demography, family structure, language Additional 2011 long form topics: education, ethnicity, income, immigration, mobility… NHS sample size is 4.5 million dwellings (f = 30%) 3 1. Introduction Non-response error in the NHS: • Survey now voluntary => expect significant non-response • To minimize the impact, after a fixed date restrict the collection efforts to a Non-Response Follow-Up (NRFU) random sub-sample U sr snr s NRFU NRFUr NRFUnr Set-up developed by Hansen & Hurwitz (1946) 1. 2. 3. 4. 4 Select 1st phase sample s from population U Non-response snr observed in s NRFU selected from snr Response NRFUr and non-response NRFUnr observed in the NRFU (HH assumed 100% resp. rate) 1. Introduction U sr snr NRFU NRFUr 5 s NRFUnr When 100% of the NRFU responds (as in Hansen and Hurwitz original setting), the NRFU can be used to estimate without non-response bias the total in snr This is not the case in the NHS. However focusing the collection efforts on the NRFU converts part of the non-response bias (that would be observed in the full snr) into sub-sampling error 2. Handling non-response error The estimation method chosen to minimize the remaining non-response bias should have the following properties: • As few bias assumptions as possible should be made • The method should be simple to explain and to implement in production Available micro-level auxiliary data to adjust for nonresponse: • 2011 Census short form • Tax data Calibration: Agreement with Census totals is desirable from a user’s perspective 6 2. Handling non-response error First class of contenders: Reweighting • Usual method used to compensate for total non-response in social surveys • The Hansen & Hurwitz estimator of a total ˆtHH yk sr ak NRFU yk ak k s nr is unbiased if 100% of the NRFU answers When the assumption does not hold, we must model the last non-response mechanism/phase and reweight accordingly… 7 2. Handling non-response error Scores method: • Model the probability of response with a logistic regression • Form Response Homogeneity Groups (RHG) of respondents and non-respondents with similar predicted response probabilities • Calculate the response rate in each RHG and assign these new predicted response probabilities to respondents • Divide the NRFUr weights by this probability: ˆtscores yk sr 8 ak NRFU r yk ak k s pˆ kRHG nr 2. Handling non-response error Second class of contenders: Imputation • Usual method to compensate for item non-response • We will consider nearest-neighbour imputation using the CANadian Census Edit & Imputation System (CANCEIS) only 1. Partial imputation: Impute only non-respondents to the subsample (NRFUnr) and use reweighting to take sampling into account ˆtpartial yk sr ak yk NRFU r ak k s nr NRFU nr yˆk ak k s nr 2. Mass imputation: Impute all non-respondents (snr/NRFUr) tˆmass 9 sr NRFU r yk ak snr NRFU rc yˆ k ak 2. Handling non-response error Some pros & cons Method Scores Preserves micro-level information of non-respondents 10 Partial imputation Mass imputation √ √√ Does not create synthetic information √√ √ Uses less heavy non-response hypotheses √√ √√ Fully takes sub-sampling design into account √√ √√ Census systems available √√ √√ More calibration to known Census totals can be done √ √√ 3. Simulation set-up Use 2006 Census 20% long form sample data Restricted to Census Metropolitan Area (CMA) of Toronto Simulation aimed at preserving the properties of the NHS (except for the f = 30%): • Non-response to the 1st phase was simulated by deterministically blanking out the data of the 63% of respondents who answered last in 2006 • Of these non-respondents, the 78% who answered first will have their response restored if they are selected in the NRFU sub-sample • NRFU sub-sampling was simulated by selecting a stratified random sample of 41% of snr 11 3. Simulation set-up Estimators calculated • As points of reference, unbiased estimators: ˆt2006 yk s ˆtHH yk ak ak sr NRFU yk ak k s nr • As contenders: ˆtscores yk sr ak tˆmass 12 ak sr NRFU r ak k s pˆ kRHG NRFU r ˆtpartial yk sr yk nr yk NRFU r yk ak ak k s nr snr NRFU rc NRFU nr yˆ k ak yˆk ak k s nr 3. Simulation set-up The scores method • A single logistic regression was done for the whole CMA of Toronto • Household response probability was predicted • Considered for stepwise selection: household-level variables, our best attempt at summarizing the person-level information and one paradata variable • R-square of 26% • 13 RHG formed with predicted probabilities ranging from 29% to 95% 13 3. Simulation set-up Imputation methods • Nearest-neighbour imputation done with CANCEIS • RHG is defined by household size • The distance between non-respondents and donors (respondents) is defined by weighting each household-level, person-level and paradata characteristics in the distance function • Preference is given to donors who are geographically close • For each non-respondents, a list of donors is made and one is randomly selected with probability proportional to a measure of size (1st phase weight for mass imputation, score method weights for partial imputation) 14 3. Simulation set-up M=84 non short form characteristics over the various topics Average relative difference: • Calculated at the CMA level: 100 M tˆj tˆ2006 j M j 1 tˆ2006 j 100 M tˆj tˆHHj M j 1 tˆHHj • At the Weighting area (953 WA in total) level within the CMA: 100 953M 15 953 M i 1 j 1 tˆij tˆ2006ij tˆ2006ij 100 953M 953 M i 1 j 1 tˆij tˆHHij tˆHHij 4. Results Errors at the CMA and WA levels for Toronto CMA WA Point of comparison Point of comparison Full firstphase Hansen & Hurwitz estimator Mass imputation Partial imputation Scores method 16 Hansen & Hurwitz Full firstphase Hansen & Hurwitz 0.94 0.00 22.98 0.00 2.97 N/A 24.56 N/A 2.25 1.52 26.69 13.22 2.03 1.45 26.77 18.67 5. Limits of the study Results: • The simulation only includes one replication of the subsampling and non-response mechanisms • Non-response bias is the measure of interest, but errors were presented • Non-response mechanisms were generated deterministically. Should they be generated probabilistically? • The 2011 sampling, non-response and available data (ex: paradata) cannot be replicated exactly • Only totals studied. What about other parameters such as correlations? 17 5. Limits of the study Possible confounding effects: • Logistic regression was done at the aggregated level of the CMA and no WA effect or interaction were considered • Paradata for imputation is more closely related to nonresponse mechanism (give preference to late respondents in the distance) • Weighting of donors in imputation has an impact • Calibration done from sample to U; calibration at inner levels/phases could help scores and partial imputation 18 6. Conclusion With these preliminary results, it seems scores method is doing well at aggregate levels, while partial imputation is doing better than scores at finer levels • • • 19 Mass imputation: Can you override the known sub-sample design with an imputation model? Partial imputation: Can include more information (personlevel, paradata) than scores, but weighting of each component in the distance is partially data driven and not straightforward Scores method: More difficult to include the information, but variable selection to explain non-response is direct 7. Future Work Possible: • • • Definite: • • 20 Replicate sub-sampling and imputation more than once to isolate bias components Consider other levels of calibration in the comparisons Hybrid of scores and partial imputation Implement a method into NHS production Estimate the errors and variances (multi-phase, large sampling fractions, errors due to modeling,…) and educate data users Important to get a good model for the last nonresponse mechanism. Whatever the method, quality of the results is a function of the auxiliary data available. For more information, please contact: François Verret - SSMD/DMES [email protected] (613) 951-7318 21