Report

Ch 6: Making Sense of Statistical Significance: Decision Errors, Effect Size, and Power Pt 2: Sept. 26, 2013 Statistical Power • Probability that the study will produce a statistically significant result when the research hypothesis is in fact true – That is, what is the power to correctly reject the null? – Upper right quadrant in decision table – Want to maximize our chances that our study has the power to find a true/real result • Can calculate power before the study using predictions of means – or after study using actual means Statistical Power • Steps for figuring power: 1. Gather the needed information: (N=16) * Mean & SD of comparison distribution (the distrib of means from Ch 5 – now known as Pop 2) * Predicted mean of experimental group (now known as Pop 1) * “Crashed” example: Pop 1 “crashed group” mean = 5.9 Pop 2 “neutral group/comparison pop” μ = 5.5, = .8, m = sqrt (2)/N m = sqrt[(.8 2) / 16] = .2 Statistical Power 2. Figure the raw-score cutoff point on the comparison distribution to reject the null hypothesis (using Pop 2 info) • For alpha = .05, 1-tailed test (remember we predicted the ‘crashed’ group would have higher fault ratings), z score cutoff = 1.64. • Convert z to a raw score (x) = z(m) + μ x = 1.64 (.2) + 5.5 = 5.83 • Draw the distribution and cutoff point at 5.83, shade area to right of cutoff point “critical/rejection region” Statistical Power 3. Figure the Z score for this same point, but on the distribution of means for Population 1 (see ex on board) • That is, convert the raw score of 5.83 to a z score using info from pop 1. – Z = (x from step 2 - from step 1exp group) m (from step 1) – (5.83 – 5.9) / .2 = -.35 – Draw another distribution & shade in everything to the right of -.35 Statistical Power 4. Use the normal curve table to figure the probability of getting a score higher than Z score from Step 3 • • • Find % betw mean and z of -.35 (look up .35)… = 13.68% Add another 50% because we’re interested in area to right of mean too. 13.68 + 50 = 63.68%…that’s the power of the experiment. Power Interpretation • Our study (with N=16) has around 64% power to find a difference between the ‘crashed’ and ‘neutral’ groups if it truly exists. – Based on our estimate of what the ‘crashed’ mean will be (=5.9), so if this is incorrect, power will change. – In decision error table 1-power = beta (aka…type 2 error), so here: – Alpha? – Power? – Beta? Influences on Power • Main influences – effect size & N • 1) Effect size – bigger d more power – Remember formula: 1 2 d – Bigger difference between the 2 group means, more power to find the difference (that difference is the numerator of d) – Also, the smaller the population standard deviation, the bigger the effect size (sd is the denominator) (cont.) • Figuring power from predicted effect sizes – Sometimes, don’t know 1 for formula, can estimate effect size instead (use Cohen’s guidelines: .2, .5, .8 or -.2, -.5, -.8) Predicted1 2 (d )() Example: Practical Ways of Increasing the Power of a Planned Study • Rule of thumb: try for at least 80% power – Interpretation of 80% power – we have a .8 probability of finding an effect if one actually exists • See Table • 1) Try to increase effect size before the experiment (increase diffs betw 2 groups) – Training/no training group – how could you do this? • 2) Try to decrease pop SD – use standardization so subjects in 1 group receive same instructions • 3) Increase N • 4) Use less stringent signif level (alpha) – but trade-off in reducing Type 1 error, so usually choose .05 or .01. • 5) Use a 1-tailed test when possible