Effectiveness of Simulation in Healthcare Joseph Barton, MD, MHMS September 18, 2014 Desired Benefits of Simulation Training • Reduce patient risk • Broadly applicable and widely available • Consistent training and assessment • Reduce costs • Provide a clinical benefit Kirkpatrick’s 4 Levels of Learning Evaluation – Reaction – Learning – Behavior – Results Learning Evaluation Level Description Evaluation 1 Surveys Did the trainee like it? 2 Assessments Did the trainee learn? 3 Transfer Is a behavior changed? 4 Benefit Is there a clinical outcome? DOES SIMULATION-BASED EDUCATION WORK? What is the Evidence? • How effective is simulation as a modality? • Number of recent meta analyses – Virtual patients – “Technology enhanced” simulation • Virtual patients do not require special equipment • Standardized patients aren’t well studied HOW EFFECTIVE IS SIMULATION AS A TEACHING MODALITY? Effectiveness of Simulation • More than 600 studies comparing 36K plus subjects • Three recent large meta analyses Effectiveness of Simulation • Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: a systematic review and metaanalysis. Acad Med. 2010;85:1589-1602. • Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978-988. • Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical education: a systematic review. J Gen Intern Med. 2013. Effectiveness of Simulation • Virtual patients and technology enhanced simulation show consistent, large and statistically significant benefits in – Knowledge – Instructor ratings – Computer scores – Patient care behaviors Direct Patient Effects? • Airway management, endoscopy, CVC insertion studied – Smaller benefits, still significant (major complications, mortality, length of stay) when compared to no intervention – Non simulation instruction did not reach statistical significance Simulation based education is more effective than no instruction. HOW DOES SIMULATION COMPARE TO OTHER INSTRUCTION? Simulation vs other intervention • Over 100 studies and 7K participants • Challenging question due to different simulation interventions and due to variable comparison strategy Simulation vs Other Intervention • Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness of technologyenhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc. 2012;7:308-320. • Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and metaanalysis. Med Teach. 2013;35:e844-e875. • Lorello GR, et al. Simulation-based training in anesthesiology: a systematic review and meta analysis. Br J Anaesth. 2014 Feb; 112(2):231-45. • McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., & Wayne, D. B. (2011). Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Academic Medicine, 86, 706–711. Simulation vs Other Intervention • Simulation is non inferior to other approaches • Technology enhanced simulation shows small, statistically significant benefit for knowledge and skills outcomes • Approached statistical significance with direct patient benefit and provider behaviors Simulation vs Other Intervention • Anesthesia review showed moderate effects for satisfaction and skills, large effect for provider behavior, small effect for direct patient benefit compared with non simulation instruction • Negligible effects comparing simulation to alternative simulation interventions • Analysis showed inconsistencies in measurement of non technical skills Pooled effect sizes for studies comparing training with no training. Effect sizes represent Cohen’s d or the nearly-equivalent Hedges’ g from random-effects meta-analysis; > 0.80 is large, 0.50–0.79 is moderate. Data derived from metaanalyses of Internet-based instruction (Cook et al. 2008b) virtual patients (Cook et al. 2010a), simulation-based instruction (McGaghie et al. 2011), and simulation-based instruction (Cook et al. 2011a) Simulation based education is probably as good as (but is not necessarily better than) other types of instruction. There is a high degree of variability between studies, suggesting that certain simulation interventions may be more effective in certain scenarios. HOW CAN WE IMPROVE THE EFFECTIVENESS OF SIMULATION TRAINING? Comparative Effectiveness • Evaluates outcomes and processes leading to the outcome – Considers the costs, barriers, unforeseen consequences, and effective strategies associated with implementing therapies in practice. • Studies comparing different simulation based approaches to explain what works, for what audience and in what context. Comparative Effectiveness • Sample sizes must be large • Confounding can be a problem • Effect size? – The difference between teaching and no teaching should be large, differences between teaching types may not be as apparent Comparative Effectiveness • Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulationbased education: systematic review and meta-analysis. Med Teach. 2013;35:e844-e875. Comparative Effectiveness • 289 studies, 20K participants • Feedback, repetition, range of difficulty, cognitive interactivity, clinical variation, distributed practice, individualized training, and longer training time significantly improve skill outcomes • Patient outcome analysis revealed benefits of similar direction and magnitude that approached statistical significance Future research should clarify how to choose between simulation and nonsimulation approaches using rigorous qualitative studies that explore the strengths and appropriateness of each approach. IS SIMULATION TRAINING COST EFFECTIVE? What is the Value? • Must consider costs – Simulator, faculty time, training expenses, facility fees, opportunity cost • No study has offered a complete accounting of simulation costs Cost Assessment • Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. Cost: the missing outcome in simulation-based medical education research: a systematic review. Surgery. 2013;153:160-176. • Norman G, Dore K, Grierson L. The minimal relationship between simulation fidelity and transfer of learning. Med Educ. 2012;46:636-647. What is the Value? • Cost reporting is infrequent and incomplete • Low-fidelity, low-cost training models can yield outcomes equal to much more expensive simulators More expensive simulators are not necessarily better. CONCEPTUALLY, SIMULATION TRAINING MAKES SENSE Simulation Training = Patient Safety? • Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Acad Med. 2003;78:783-788. Simulation Education = Patient Safety? • “Risk free” skill rehearsal – Repetitive, deliberate and structured practice • “Risk free” assessment and feedback Simulation based rehearsal is evidence supported when learning to work with “real patients.” SIMULATION MAY NOT ALWAYS BE THE BEST LEARNING MODALITY Is Simulation the Best Choice? • Instructional design should be based on learning objectives, learner needs, safety concerns, resource utilization • Virtual patients often used to teach clinical reasoning • Technology enhanced simulation is most often for procedural training How Do We Optimally Implement Simulation Education? • More research to clarify how to choose between sim and non sim approaches • Cost effectiveness research to determine true and comparison value of sim education How Do We Optimally Implement Simulation Education? • Comparative effectiveness research should focus on what works in simulation – for which audience – for what circumstances – and at what cost. • Task analysis should focus on critical actions and can help determine level of fidelity necessary for type of training How Do We Optimally Implement Simulation Education? • Careful selection and sequencing of events surrounding the simulated task • Appropriate faculty development and support • Institutional commitment to simulation QUESTIONS?