Report

Effect Size Issues Rob Coe WRI Workshop, 18 March 2013 Four parts I II What is Effect Size? The case for using effect size • III Problems in using effect size • 5 reasons) (6 problems) IV Recommendations • (13 recommendations) 2 What is Effect Size? 3 © 2003 Robert Coe, University of Durham Sources Coe, R. (2002) It's the effect size, stupid: what effect size is and why it is important. Paper presented at the Annual Conference of the British Educational Research Association, University of Exeter, England, 12-14 September 2002. Coe, R.J. (2012) ‘Effect Size’ in J. Arthur, M. Waring, R. Coe, and L.V. Hedges (Ed.s) (2012) Research Methods and Methodologies in Education. London: Sage. 4 © 2013 Robert Coe, University of Durham Normal distribution High standard deviation (spread out) Low standard deviation (tightly grouped) Effect Size is the difference between the two groups, relative to the standard deviation Mean of experimental group – Mean of control group Effect Size = Standard deviation Examples of Effect Sizes: ES = 0.2 58% of control group below mean of experimental group “Equivalent to the difference in heights between 15 and 16 year old girls” Probability you could guess which group a person was in = 0.54 Change in the proportion above a given threshold: from 50% to 58% or from 75% to 81% ES = 0.5 69% of control group below mean of experimental group “Equivalent to the difference in heights between 14 and 18 year old girls” Probability you could guess which group a person was in = 0.60 Change in the proportion above a given threshold: from 50% to 69% or from 75% to 88% ES = 0.8 79% of control group below mean of experimental group “Equivalent to the difference in heights between 13 and 18 year old girls” Probability you could guess which group a person was in = 0.66 Change in the proportion above a given threshold: from 50% to 79% or from 75% to 93% Effect Sizes from EEF Toolkit 11 © 2013 Robert Coe, University of Durham The case for using effect size measures 12 © 2003 Robert Coe, University of Durham Source Coe, R. (2004) ‘Issues arising from the use of effect sizes in analysing and reporting research’ in I. Schagen and K. Elliot (Eds) But what does it mean? The use of effect sizes in educational research. Slough, UK: National Foundation for Educational Research. http://www.nfer.ac.uk/nfer/publications/SEF01/SEF01.pdf 13 © 2013 Robert Coe, University of Durham 1. Effect size enables uncalibrated measures to be interpreted From a questionnaire on teachers’ perceptions of their training needs (7 item scale, each item coded 1-4) By age group: age 20-40 mean 2.98 n 389 By gender: age 41-65 SD 0.87 mean 2.09 n 345 female SD 0.95 “younger teachers expressed stronger needs than their older colleagues” Effect size = 0.98 mean 2.64 n 451 male SD 1.05 mean 2.44 n 283 SD 1.11 “female teachers appeared to have higher training needs than males” 14Effect © 2003 Robert Coe, University of Durham size = 0.19 2. Effect size emphasises amounts, not just statistical significance The dichotomous “significant/not” decision is almost never appropriate The size of a difference is almost always important “Significance” has many meanings, but is inevitably related to the size of the difference 15 © 2003 Robert Coe, University of Durham Nonsensical dichotomies Experimental Group 52 69 83 66 58 69 68 44 62 51 mean SD 31 70 44 63 70 55 86 74 68 74 63.0 13.7 Control Group 51 60 57 80 45 37 56 47 55 63 mean SD Experimental Group 51 45 69 62 63 46 73 39 49 52 52 69 83 66 58 69 68 44 62 51 54.9 11.2 mean SD t-test gives: p= 0.049 31 70 44 63 70 55 86 74 68 74 63.0 13.7 Control Group 51 60 57 80 45 37 56 47 55 63 mean SD 51 45 70 62 63 46 73 39 49 52 55.0 11.3 t-test gives: p= 0.052 Statistically significant difference Difference not significant THE TREATMENT WORKED! IT DIDN’T WORK! 16 Don’t ignore amounts 3 2.5 2 1.5 1 0.5 0 -0.5 -1 (a) "not significant" (b) "significant" 17 © 2003 Robert Coe, University of Durham (c) "significant" Types of significance Statistical significance o Probability that difference due to chance Practical significance o Theoretical or applied importance Clinical significance o ‘extent to which the intervention makes a real difference to the quality of life’ (Kazdin, 1999) Economic significance o Benefit in relation to cost (Leech and Onwuegbuzie, 2003) 18 © 2003 Robert Coe, University of Durham 3. Effect size draws attention to the margin of error Statistical power is important, but often overlooked Much apparent disagreement is actually just sampling error 19 © 2003 Robert Coe, University of Durham 3 2.5 2 1.5 1 0.5 0 -0.5 -1 20 © 2003 Robert Coe, University of Durham 4. Effect size may help reduce reporting bias The “file-drawer” problem is alive and well Within-study reporting bias can also be a problem 21 © 2003 Robert Coe, University of Durham 3 2.5 2 1.5 1 0.5 0 -0.5 -1 22 © 2003 Robert Coe, University of Durham 5. Effect size allows the accumulation of knowledge Meta-analysis can combine results from different studies Small studies are worth doing 23 © 2003 Robert Coe, University of Durham Problems in using effect size measures 24 © 2003 Robert Coe, University of Durham 1. Which effect size? Proportion of variance accounted for o Universal measure, but o Non-directional o Sensitive to violations of assumptions o Large standard errors o Interpretation counter-intuitive o ‘Effect’ should mean effect Non-parametric effect size measures Odds ratio Un-standardised (raw) difference 25 © 2003 Robert Coe, University of Durham 2. Which standard deviation? Pooled or control group? o Control group is conceptually purer o Pooled is statistically better (provided compatible) o Sometimes there isn’t a ‘control’ group Residual standard deviation o Residual gain becomes ‘progress’ o Effect sizes substantially inflated and dependent on correlation o Important to report clearly Restricted range o Effect size higher if range limited 26 © 2003 Robert Coe, University of Durham 3. Measurement reliability Standardised mean difference is spuriously affected by the reliability of the outcome measure o Part of the variance in measured scores is due to measurement error o More error Higher S.D. Lower E.S. o Reliability should be reported 27 © 2003 Robert Coe, University of Durham 4. Non-normal distributions -4 -3 -2 -1 0 28 © 2003 Robert Coe, University of Durham 1 2 3 4 -4 -3 -2 -1 0 29 © 2003 Robert Coe, University of Durham 1 2 3 4 An effect size of 1 Normal -3 -2 -1 Contaminated-normal 0 1 2 3 4 -3 Median person raised to the 84th percentile -2 -1 0 1 2 3 4 5 Median person raised to the 97th percentile 30 © 2003 Robert Coe, University of Durham 6 5. Interpreting effects Cohen’s 0.2 = small 0.5 = medium 0.8 = large is a bit simplistic Interpretation depends on o Translation into familiar metric o Comparison with known effects o Costs o Feasibility o Benefits (and their value) o Availability of alternatives 31 © 2003 Robert Coe, University of Durham 6. Incommensurability Commensurable outcomes o Construct o Operationalisation o Reliability Commensurable treatments o Well defined? o Fidelity of delivery o Intensity / duration o Control group treatment Commensurable populations o Range 32 © 2003 Robert Coe, University of Durham Recommendations 33 © 2003 Robert Coe, University of Durham Recommendations 1. Calculate and report standardised effect size, with confidence interval / standard error, for all comparisons 2. Show these graphically 3. Report all relevant comparisons regardless of whether confidence intervals include zero 4. Interpret effect sizes by comparison with known effects and in relation to familiar metrics 5. Report un-standardised raw differences whenever the outcome is measured on a familiar scale 34 © 2003 Robert Coe, University of Durham Recommendations (cont) 6. Interpret the significance of an effect with regard to issues such as its o o o o o o o effect size theoretical importance associated benefits associated costs policy relevance feasibility comparison with available alternatives 7. Don’t use the word ‘effect’ (with or without ‘size’) unless a causal claim is intended and can be justified 35 © 2003 Robert Coe, University of Durham Recommendations (cont) 9. Be cautious about the calculation and interpretation of standardised effect sizes whenever o Sample has restricted range o Population is not known to be normal o Outcome measure has low or unknown reliability o Outcomes have been statistically adjusted (residuals) 10.Always report reliability of measures, extent of restriction, correlations (or R2) in these cases 36 © 2003 Robert Coe, University of Durham Recommendations (4) 11.Small studies with low power and statistically non-significant effects should still be conducted, reported and published, provided they are free from bias 12.Synthesise the results of compatible studies using meta-analysis 13.Beware of combining or comparing effect sizes from studies with incommensurable outcomes, treatments or populations 37 © 2003 Robert Coe, University of Durham