Report

Effect Size & Power Analysis 0246-511 Statistics for Graduate Study I 1 Learning Objectives Understand common effect size measures Which to use When to use different types Learn to employ effect sizes to plan study A priori power analysis Two groups ANOVA Observed power 2 Effect Size Two Broad Types (in ANOVA) Standardized mean differences Proportion of variance accounted for Compared to statistical significance APA Guidelines Use in Meta-Analysis 3 Effect Size: Comparing Two Means Standardized Mean Differences Population Value Estimated from sample 1 2 12 error X1 X 2 d MSW Proportion of Variance Accounted for 2 t r 2 t df 2 4 Concepts (Two-Groups) Power Type I & Type II Error Effect Size γ (gamma) δ (delta) Estimating Power Estimating Needed Sample Size Observed Power 5 Review Power Type I Error Type II Error Factors that affect power 6 Independent Samples t-test: Things we need to estimate power Estimate Effect Size Estimate Gamma Estimate Delta Estimate Power Working backwards to get N 7 An example Fire Safety Instruction Two methods of teaching Pilot Study Need to estimate sample size needs to get funding Power of .80 at alpha .05 and .01 Score 1 X1 77 X2 77 2 3 4 67 60 79 80 81 82 5 6 7 8 65 71 68 77 60 70 74 74 9 10 69 80 77 83 8 We will need to Estimate of Effect Size Gamma Estimate Delta 1 2 x1 x2 ,d sp Estimate (observed) Power Work backwards to get n n 2 n 2 2 2 9 Results from Pilot Study Score 1 2 3 4 5 6 7 8 9 10 Mean Sum SS Std Dev Variance pooled var std err t X1 77 67 60 79 65 71 68 77 69 80 71.30 713.00 402.10 6.68 44.68 46.09 3.04 -1.48 X2 77 80 81 82 60 70 74 74 77 83 75.80 758.00 427.60 6.89 47.51 71.3 75.8 0.66 46.09 10 0.66 1.48 2 Observed Power (.05) ≈ 0.32 Observed Power (.01) ≈ 0.14 Note: strictly speaking, all we need to estimate n for power of .80 is gamma, we compute observed power for illustration and because it is sometimes helpful. 10 Obtaining estimate of n 2(2.802 ) n.05 36.0 2 0.66 2(3.402 ) n.01 53.08 2 0.66 Note1: these are not obtained delta estimates, but based on what is needed to obtain desired power level Note 2: these are “n” values not “N” values, thus need this many per condition. 11 Unequal Sample Sizes For unequal sample sizes we often use something called a harmonic mean. It is calculated as follows: k Xh 1 Xi If our group sizes had been 14 and 9: Xh 2 1 1 14 9 11.11 12 Effect Sizes: More than two means Population Based Measure 2 2 2 total 2 error total Estimated from Sample SSM (k 1) MSW SSTOT MSW 2 Another sample measure SS M SStot 2 Also called R2 13 A brief digression ω2 can be derived from the F statistic as well: (k 1)( F 1) (k 1)( F 1) kn 2 You will care shortly. 14 Cohen’s Effect Size Estimate Cohen defines a population ANOVA ES estimate as: M f Stevens provides an estimate of this as: (k 1) F f N ^ Keppel & Wickens provide another estimate: 2 f 1 2 ^ 15 Extend previous example Score 1 2 3 4 5 6 7 8 9 10 Mean SS Source Method Within Total X1 77 67 60 79 65 71 68 77 69 80 71.3 402.1 X2 77 80 81 82 60 70 74 74 77 83 75.8 427.6 X3 79 74 69 70 71 63 69 69 72 75 71.1 166.9 SS 141.267 996.600 1137.867 df 2 27 29 MS 70.633 36.911 2 141.267 (3 1)36.911 .06 1137.867 36.911 F 1.914 0.167 16 Use power chart to obtain a noncentrality paramter: Φ 17 Using power charts to get sample size What is a noncentrality parameter? From examining a power chart, Φ≈1.8 Now we substitute our obtained Φ value into the following equation: n 2 1 2 2 1 .06 n 1.8 50.76 .06 2 Rounding up to 51, this implies 3(51-1)=150 for dfdenom. We need to iterate again. Using the infinity line, we get… 1 .06 n 1.83 52.44 .06 2 18 Strategy for assessing power Determine form of experiment/research Decide on hypothesis test Carefully consider which effect(s) is (are) important to detect Select desired power (& alpha) Determine likely effect size (or range of effect sizes) Conduct power analysis Consider whether study is feasible as currently designed 19 In-Class Example Students’ (MA Thesis) involves trying to find a hypothesized effect. Wish to detect effect it if it is at least ω2=.03 df numerator will be 1 How many subjects will she have to recruit? 20 Software Options SPSS (observed power) G*Power (consistent with Dr. Jackson’s 2nd principle of economics) Web-based power calculators Others 21