Report

Computing Random Effects Models in Meta-analysis Terri Pigott, C2 Methods Editor & co-Chair Professor, Loyola University Chicago tpigott@luc.edu The Campbell Collaboration www.campbellcollaboration.org As I am going through the beginning slides: • Please download RevMan if you do not already have it: – http://ims.cochrane.org/revman/download • Data is here: https://my.vanderbilt.edu/emilytannersmith/training-materials/ Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Fixed and Random Effects Models in Meta-analysis • How do we choose among fixed and random effects models when conducting a meta-analysis? • Common question asked by reviewers working on systematic reviews that include a meta-analysis Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Our goal today • Provide a brief description of fixed and of random effects • • • • models Discuss how to estimate key parameters in the random effects model Do sample computations to illustrate how to obtain the variance component when computing the mean effect size Illustrate how to compute the random effects model in RevMan Share resources for these methods for reviewers Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org An advertisement • Much of my own thinking on these issues has been clarified by the following sources: Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed effect and random effects models for meta-analysis. Research Synthesis Methods, 1, 97-111. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley. Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Our choice between the two models depends on: • Our assumption about how the effect sizes vary in our meta- analysis • The two models are based on different assumptions about the nature of the variation among effect sizes in our research synthesis Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Fixed-Effect model • Borenstein et al. adopt the wording of fixed effect (no “s”) here because in a fixed effect model, we assume that the effect sizes in our meta-analysis differ only because of sampling error and they all share a common mean • Our effect sizes differ from each other because each study used a different sample of participants – and that is the only reason for the differences among our estimates Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Picture from Borenstein et al. (2010) Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org In a fixed-effect model • Note that the effect size from each study estimate a single common mean – the fixed-effect • We know that each study will give us a different effect size, but each effect size is an estimate of a common mean, designated in the prior picture as θ Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org In a random effects model • We assume two components of variation: – Sampling variation as in our fixed-effect model assumption – Random variation because the effect sizes themselves are sampled from a population of effect sizes • In a random effects model, we know that our effect sizes will differ because they are sampled from an unknown distribution • Our goal in the analysis will be to estimate the mean and variance of the underlying population of effect sizes Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Another picture from Borenstein et al. (2010) Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org In a random effects model • We see in the picture that each distribution has its own mean that is sampled from the underlying population distribution of effect sizes • That underlying population distribution also has its own variance, τ2 , commonly called the variance component • Thus, each effect size has two components of variation, one due to sampling error, and one from the underlying distribution Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Some notation for shorthand vi2 is the sampling variance for our effect size is the variance component, 2 the variance of our effect size distribution Ti is our effect size (SMD, odds-ratio, correlation, etc) is the mean of the underlying effect size distribution Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org In fixed effects, we can write our model as Ti ei , where 2 i ei ~ N (0, v ) So, each effect size estimates a single mean effect, θ, and differs from this mean effect by sampling error Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org In random effects, we can write our model as Ti ei i , where ei ~ N (0, v ) and i ~ N (0, ) 2 i 2 Each effect size differs from the underlying population mean, μ, due to both sampling error and the underlying population variance Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Given the prior slides, how do we choose? • The fixed-effect model assumes only sampling error as the source of variation among effect sizes • This assumption is plausible when our studies are close replications of one another, using the same procedures, measures, etc. Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Choosing a model (continued) • In the random effects model, we assume that our effect sizes are sampled from an underlying population of effect sizes • We already assume that our studies will differ not only because there are different participants, but also because of differences in the way studies were conducted • Thus, we often choose random effects models because we anticipate variation among our studies for a number of reasons Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Implications of the choice • Recall that all our analyses in a meta-analysis are weighted by the inverse of the variance of the effect size, i.e., by the precision of the effect size estimate • For a random effects model, the variance for each effect size is equal to vi + τ2 • In other words, in a fixed effect model, we will more heavily weight larger studies. In a random effects model, the larger studies will not be weighted as heavily Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Example of a fixed effect analysis from RevMan Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Same data, random effects analysis Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org So, where are we now? • We have discussed the underlying assumptions of fixed effect and random effects models • We have also talked about how we choose between the two models • Now we will talk about how to estimate the parameters of a random effects model Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Review of steps for fixed effect basic meta-analysis • Compute the effect size and its variance for each study • Compute the study weight – the inverse of the variance (1/var) • Compute the weighted mean effect size and its standard error (inverse of the sum of the weights) • Compute the 95% confidence interval for the weighted mean • Compute the test of homogeneity Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Goals in a basic meta-analysis using a random effects model • We want to estimate a mean effect size, μ, the mean effect size from the underlying population • We also want to estimate the variance of the underlying effect size distribution, τ2 • Biggest difficulty: how to estimate τ2 Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Estimating the variance component, τ2 • There are two main methods for computing τ2 – Variously called the method of moments, the DerSimonian/Laird estimate – Restricted maximum likelihood • The method of moments estimator is easy to compute and is based on the value of Q, the homogeneity statistic • Restricted maximum likelihood requires an iterative solution Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Method of moments estimator Q (k 1) , where c Q is the homogeneity statistic, k is the number of effect sizes, and 2 c is based on the fixed effect weights, wi 1/ vi , and is c w i 2 w i w i This is the estimator used in many meta-analysis programs such as RevMan and CMA Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org An example • So before we do anything in our basic random effects meta- analysis, we need to compute Q and the fixed effects weights, wi = 1/vi , to get the variance component, τ2,. • Once we have our τ2 , we can continue with our computation of the mean effect size and its variance. • We will use an example of a meta-analysis of gender differences in transformational leadership Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Eagly, Johannesen-Schmidt & van Engen (2003) • This synthesis examines the standardized mean difference estimated in primary studies for the difference between men and women in their use of transformational leadership. • Transformational leadership involves “establishing onself as a role model by gaining the trust and confidence of followers” (Eagly et al. 2003, p. 570). • The sample data is a subset of the studies in the full metaanalysis, a set of 24 studies that compare men and women in their use of transformational leadership Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Data Study name Bass 1996-3 Bono & Judge 2000 Boomer 1994 Carless 1998 Church & Waclawski 1998 Church & Waclawski 1999 Church et al 1996 Cuadrado 2002 Daughtry & Finch 1997 Davidson 1996 Gillespie & Mann 2000 Hill 2000 Jantzi & Leithwood 1996 Judge & Bono 2000 Komives 1991 Kuchinke 1999 Leithwood & Jantzi 1997 Manning 2000 McGrattan 1997 Rosen 1993 Rozier & Hersh-C 1996 Sosik & Megerian 1999 Spreitzer et al. 2000 Wipf 1999 Campbell Collaboration Colloquium – May 2012 Effect size -0.09 -0.1 -0.62 -0.17 0.61 0.2 -0.22 -0.17 -0.25 -0.15 -0.1 -0.36 -0.29 -0.13 0.31 0.09 -0.35 -0.08 -0.72 -0.44 -0.36 0 -0.21 -0.87 SE eff size N men 0.066436 420 0.148128 112 0.262187 30 0.083113 368 0.091648 456 0.150202 1236 0.117768 209 0.185405 65 0.147431 130 0.280935 27 0.252084 92 0.356387 29 0.104781 288 0.117224 134 0.077847 296 0.214466 73 0.067508 965 0.199267 44 0.241996 42 0.224536 29 0.087466 229 0.283473 112 0.089504 326 0.239102 29 N women 493 77 31 240 50 132 111 53 72 24 19 11 135 160 383 31 288 59 32 67 316 14 204 55 www.campbellcollaboration.org Back to our Method of moments estimator Q (k 1) , where c Q is the homogeneity statistic, where 2 Q 2 2 w T ( w T ) i i i / wi k is the number of effect sizes, and c is based on the fixed effect weights, wi , and is c w i 2 w i w i So we need c, and Q Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org To get c: ∑ wt - ∑ wt2/ ∑ wt = 1695.922 – (223238.2)/1695.922 = 1596.29 To get Q: 167.151-(-186.795)2 /1695.922 = 146.58 To get τ2 : (Q – (k-1))/c = [146.58 – (24-1)]/1596.29 = 0.079 Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Now that we have τ2 , we need to get the mean and its variance • We first need to add τ2 to each study’s variance to get the random effects variance • Then we get the new weights to use for the computation of the mean effect size and its variance Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Data The random effects mean is found from (-35.76)/223.00 = -0.16 The se for the mean is found by 1/sqrt(223.00) = 0.067 Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org RevMan for random effects models • We can also get the results in RevMan without having to do the computations. • Open the file called “Gender differences for transformational leadership.rm5” – click on open a review from a file Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Click on data and analyses (I already put in data) Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Click on 1.3 Transformational leadership score Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Click on Forest plot Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org These are the fixed effects CI’s for each study. Here is our tau and test that tau is zero. Campbell Collaboration Colloquium – May 2012 This is the random effects mean and CI www.campbellcollaboration.org Once we have the random effects mean and CI • We will want to test whether the variance component is different from zero. • Testing whether the variance component is different from zero is equivalent to asking whether the variation among studies is greater than what we would expect with only sampling error • The test: the fixed effects test of homogeneity Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Restricted maximum likelihood estimator • Many statisticians do not like the method of moments estimator though RevMan and Comprehensive Metaanalysis use this estimator • Can estimate τ2 using HLM, SAS Proc Mixed, or R • I can give you sample programs for these estimators Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Reporting guidelines for random effects models • These are the reporting guidelines according to me • You should report the following when using a random effects model for estimating a mean effect size: – Rationale for choosing the random effects model – The method for computing the variance component, τ2 (and it’s okay to say you used RevMan or CMA, for example) – The estimate of the variance component – The mean effect size and its variance or standard error – The confidence interval around the effect size Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org Some concluding remarks • Given the anticipated differences among studies in a typical social science systematic review, we usually choose random effects models • From the C2 Methods Editor (me): make sure to provide a rationale for your choice in the protocol, based on your substantive knowledge of the area of the review • Many other issues not touched on in this talk and see the references following this slide Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org References Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed effect and random effects models for meta-analysis. Research Synthesis Methods, 1, 97-111. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley. Eagly, A. H., Johannesen-Schmidt, M. C. & van Engen, M. L. (2003). Transformational, transactional, and laissez-faire leadership styles: A meta-analysis comparing women and men. Psychological Bulletin, 129, 569-591. Raudenbush, S. W. (2009). Analyzing effect sizes: Random- effects models. In Cooper, H., Hedges, L. V. & Valentine, J. C. (Eds.). The handbook of research synthesis and meta-analysis (2nd ed). New York: Russell Sage Foundation. Campbell Collaboration Colloquium – May 2012 www.campbellcollaboration.org P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway E-mail: info@c2admin.org http://www.campbellcollaboration.org The Campbell Collaboration www.campbellcollaboration.org