Report

Clinically Significant Treatment Effects: Effect Sizes and Moderators Helena Chmura Kraemer, Ph.D. Stanford University (Emerita) University of Pittsburgh Randomized Clinical Trails (RCTs) RCT: the “gold standard” method to evaluate the efficacy or effectiveness of a treatment in a population. RCT: Background A patient’s response following treatment combines: • Statistical Regression to the mean • Expectation Effects • Spontaneous changes • Secular trends • PLUS: the actual effect of treatment Thus the effect of treatment cannot be assessed in pre-post studies. RCT: The Effect of Treatment Definition of “Effect of Treatment”: • Response(T) compared to Response(not-T) Requires some control/comparison condition: “not-T”=C Requires measurement of response that is not influenced by T or C: “blindness” Problem: Cannot observe any individual patient both with T and not-T…. Thus cannot assess the effect of treatment on an individual patient. RCT: Randomization Can assess the typical effect of treatment in a population by: • Sampling the population of interest • Randomize patients to T or to C. • Compare T vs C Treatment effect on the “typical” patient on. Analysis “by intention to treat”. There is no assumption that the effect of treatment is the same for all individuals in the population. RCT: Comparisons? (Classic) “Statistical Significance” means that the design (sample size) was good enough to detect a non-random difference. • Comment on design, not on effectiveness of treatment A treatment effect may be statistically significant and clinically trivial. • Absence of “statistical significance” means Flawed conceptualization, inadequate design, measurement, sample size NOT absence of “clinical significance”. NEEDED: A measure of effect size that allows assessment of clinical significance. Powering up a RCT T<C 1.000 T=C T>C Prob significant result 0.900 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000 0.0 0.2 0.4 0.6 Effect Size, AUC 0.8 1.0 The ”critical value” or “threshold of clinical significance”? Below this effect size, clinicians and patients would consider the effect clinical trivial. The more above this value, the greater the preference for T over C. Obtained from previous research, clinical experience, opinion of experts. Effect Sizes—Recommended AUC=Prob(T>C)+.5Prob(T=C) • Null value: .5; Extremes: 0,1. SRD=Prob(T>C)-Prob(T<C)=2AUC-1 • Null value: 0; Extremes: -1, +1. Number Needed to Treat: NNT=1/SRD. How many patients would you have to treat with T to get one more success than if you had treated them with C? • Null value: infinity: Extremes: -1, +1. Effect Sizes—More Common Cohen’s d= Standardized mean difference between T and C. • Meant to be used when the responses have a normal distribution in both T and C. Almost never exact! • However, when reasonable: AUC=normsdist(d/√2) Odds Ratio—not recommended. • NNT> (√OR+1)/(√OR-1) Effect Size—Standards d AUC SRD NNT 0 Null .50 0 Infinity .2 Small .56 .11 8.9 .5 Med .64 .28 3.6 .8 Large .71 .43 2.3 1.0 .76 .52 1.9 Possible RCT Outcomes T1 vs T2: c the critical effect size: 95% Confidence Intervals for the SRD *T1 is clinically superior to T2 *T1 is non-inferior to T2 *T1 and T2 clinically equivalent. T1 and T2 clinically equivalent. A failed RCT -1 -c T1<T2 0 T1=T2 +c Effect Size (SRD) +1 T1>T2 Moderators and Mediators of Treatment in a RCT Moderator: M moderates the effect of T in a RCT on O if • M is a baseline variable (hence precedes T and O and is uncorrelated with T) • The effect size of T on O differs depending on what M is. Mediator: M mediates the effect of T in a RCT on O if • M is an event or change that happens during treatment (hence follows T but precedes O). • M is correlated with T • The effect size of T on O is explained wholly or in part by the effect of T on M. Examples: Moderators Gene (5HTT) moderates the effect of drug treatment on outcome for depressed patients (pharmacognetics). Baseline depression moderates the effect of psychotherapy for treatment of anorexia . Level of addiction moderates the effect of smoking cessation treatment on abstinence. Example 2: Exploration after a RCT. How??? Exploration Secondary Data analysis RCT Hypothesis RCT Design Publication Validation Pilot Study RCT