Using Implementation Science To guide the installation and implementation of evidence-based practice in NYC child welfare programs Children’s Justice Act and State Liaison Officers Annual Meeting, April 28-29, 2014 Hyatt Regency. New Orleans, LA PANEL • Theresa Costello, MA, Director National Resource Center for • • • • Child Protective Services David Collins, LMSW, Assistant Commissioner, Program Innovation and Planning, NYC Administration for Children's Services (ACS) Diane DePanfilis, PhD, MSW, Moses Distinguished Visiting Professor, Silberman School of Social Work at Hunter College & Professor, University of Maryland School of Social Work Besa Bauta, LSW, MPH, Senior Director of Research and Evaluation, Center for Evidence Based Implementation and Research at Catholic Guardian Services Paul Martin, LCSW-R, Director of Preventive Services, Leake and Watts Services Inc. 2 Introductions: Why are you here? 3 AGENDA for Session • Purpose of NRCCPS TA in NYC • ACS Evidence Based Practice Initiative in Child Welfare • Brief overview of Implementation Science – Illustrations and conversations about how implementation science is being used in NYC 4 A “reframe” of the Agenda through the lens of the Science of the Positive Copyright 2013 Jeff Linkenbach Evidence based practice, Implementation science TA framework Why is this Important? What & How is NYC Implementing Evidence based practice In child welfare? What does NYC expect to achieve? 5 Why is this important? Our SPIRIT We want families to be empowered to meet the basic needs of their children and to keep them safe without the need for out-of-home placement. They/we have the best chance of succeeding when we implement efficacious practices. 6 What is our science? -TA framework, evidence based practice, implementation science 7 A member of the National Training & Technical Assistance Network, a service of the Children’s Bureau, U.S. Department of Health & Human Services PURPOSE of Technical Assistance: To support the installation & implementation of EBP with families at risk of maltreatment and placement. 8 Types of TA (examples) • Outputs/Products: – Alignment activities with ACS policies – Organizational Assessment – Fidelity assessment tools – Capacity building with implementation teams and workgroups 9 Creating, Selecting, and Implementing Evidence Based Service Models David Collins Assistant Commissioner Division of Policy, Planning, and Measurement NYC Children’s Services April 2014 10 About New York City’s Administration for Children’s Services (ACS) • • • • • • Responsible for child protection, child welfare services, juvenile justice prevention and placement, and early childhood care and education in New York City. Approximately 6,000 direct FTEs. Investigates 54,000 allegations of child maltreatment each year. 25,000+ annual families served in preventive; foster care population of 11,900 (down from 49,000+ in 1991). Preventive and foster care systems are privately contracted and use a delegated case management approach. Merged with the city’s Department of Juvenile Justice in 2010 and received authority to directly oversee placement of adjudicated youth under new legislation in 2012; also provides many evidence based alternative to detention and placement diversion services. Oversees the nation’s largest publicly funded child care and early education system, serving approximately 110,000 children annually through a mix of contracted slots and vouchers. 11 New York City’s Vision for Evidence Based Programming • Goals: Building on prior experience with EBMs/PPMs, improve outcomes in juvenile justice, preventive services, foster care and parts of protective services; expand the continuum of services to better meet the needs of families. • Where EBMs and PPMs are being used in Children’s Services: • Juvenile Justice – Juvenile Justice Initiative ATP and IPAS; Family Assessment Program PINS Diversion; Behavior management/change approach in facilities; aftercare services are all EBMs/PPMs • Preventive – 3,185 slots serving up to 8,500 families per year; one third of all NYC child welfare preventive slots are now EBMs or PPMs, using a total of 11 models. Believed to be the largest and most diverse municipal EBM initiative currently in existence. • Foster Care – ChildSuccessNYC Pilot program combines functional trauma assessments, caseload reductions and EBM/PPM interventions; IV-E waiver in place to bring similar reforms online throughout regular family foster care by 2015. Also converted 126 therapeutic foster care and residential beds to Multidimensional Treatment Family Foster Care (MTFC) in 2012 in order to prevent and/or shorten the need for institutional placements. • Protective Services – Solution-Based Casework for child protective staff charged with monitoring families; also piloting in protective/diagnostic unit 12 Laying the Foundation for the NYC EBM/PPM Expansion JJI: Juvenile Justice Initiative; provided to adjudicated juvenile delinquent youth FAP: Family Assessment Program; provided as an alternative to court to families seeking a Persons In Need of Supervision (PINS) case Teen Preventive: Provided to families to prevent a foster care placement Choosing Models for Each Program/Procurement Issues Determined population and contract type (preventive; high needs foster care; residential aftercare) For solicitations that listed models, conducted national search to determine best fit Models rooted in the mental health system Models rooted in the juvenile justice system Looked at how model addresses and supports implementation with provider agencies: what do they assist with, what do they monitor? Interviewed model developers EBM open-house for ACS and providers to learn about what fits best After selection, conducted an analysis of each model to determine strengths/weaknesses/where we would need to add support Discovered procurement issues with regard to naming models Determined ACS’ “Levels of Evidence” Determined definition of “Model Developer” Defined what the minimum standards were for a “Model Manual” 14 System Readiness and Capacity Building Began with Dr. Allison Metz’s “Listening Tour” Discovered: Lack of knowledge of ACS staff; Perceived misalignment between child welfare policies and the practice of EBMs; Misalignment in the oversight of model adherence and current ACS evaluation and monitoring; Redundancy of documentation; and Limited internal training/technical assistance capacity Organized 3 Task Teams – Representatives chosen by Deputy Commissioners ACS Capacity Building Policy and Practice Alignment Evaluation and Monitoring Training Implementation Institute for ACS staff, facilitated by Dr. Allison Metz One day overviews of all models Targeted training for various ACS stakeholders: Family Court attorneys, Evaluation and Case Review staff, Program Development staff, Preventive Technical Assistance, Office of Placement, Child Protection staff Communication – all widely disseminated One pager on EBM Desk Guide of all ACS Preventive models Continuum of ACS Preventive models 15 System Readiness and Capacity Building Built Service Connect Instrument Web-based structured decision making tool 23 questions about specific caregiver, child, and family characteristics. Automatically determines their overall service level: low, moderate, high, very high. Standards/Oversight Changes Lessons Learned From Working With Various Model Developers Addenda for each evidence-based and promising practice model Monitoring tools developed for each evidence-based and promising practice model Evaluation domains reflect what’s needed for a successful EBM (e.g., fidelity tools being used; input from consultant, etc.) Models in various stages of development Focus on fidelity and fidelity instruments widely vary Different opinions on the length of time developers will be involved vs. self-sustainability Hard look at ability of government system to take on developer/consultant role Meetings, meetings, meetings – Importance of “Feedback Loops” Internal With providers all together, usually by model With individual providers With developers 16 ACS Preventive Continuum Evidence-Based, Promising Practices and Current ACS models (See separate handout) 17 Overview of Implementation Science 18 Why is this important? When we implement efficacious practices, it is important that we understand how well the new strategy is implemented to improve the likelihood that intended outcomes will be achieved. 19 Implementation Science “Children and families cannot benefit from interventions they do not experience” This is called the Implementation Gap 20 Illustrations & Conversations • Perspectives from: – Preventive service program leaders – ACS – Technical Assistance provider -- NRCCPS 21 What is “Implementation?” • A specified set of activities designed to put into practice an activity or program. • A strategic, purposeful approach, not a one-time event, for making a change. • A process for bridging the gap between “what we know” and “what we do.” 22 Implementation Science Purpose Implementation Science provides frameworks for successful implementation so that: – Children and families benefit from interventions and experience positive outcomes – Workers are supported to learn new skills, manage change, and identify barriers – Organizations are responsible for creating hospitable environments for change and supporting workers 23 Framework used by ACS NATIONAL IMPLEMENTATION RESEARCH NETWORK (NIRN) Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http://www.fpg.unc.edu/~nirn/resources/detail.cfm?res ourceID=31 24 What works to support successful implementation? • The combination of: – Stage-matched implementation activities – “Drivers” or core components that promote competency, organizational support, and leadership – Teams that provide organized capacity to lead and support the change effort 25 Stages of Implementation A 2 – 4 year cycle Full Implementation (Sustainability & Effectiveness) Initial Implementation Installation Exploration 26 Exploration & Adoption Explore the fit and feasibility of available interventions to for preventing child maltreatment and placement (Purpose of Preventive Services) SAMPLE AREAS to Explore • Review program purpose/target population • Discuss similarities & differences between current practice & new practice • Discuss fidelity criteria • Discuss installation tailoring options • Identify skills needed by staff • Examine competency building process • Understand past research CONVERSATION • What factors did you consider as you went through the exploration stage? • What process was used to make the decision to convert to an evidence based practice? • Who was involved in selecting specific model(s)? 28 Installation Collaborate on a process to individualize the implementation process SAMPLE AREAS for development • Adapt screening criteria based on target population • Develop logic model • Select standardized clinical assessment measures & arrive at core outcomes • Operationalize fidelity criteria and develop fidelity assessment procedures • Decide on model for training and ongoing consultation/coaching A Conversation • What process was used to develop a logic model? • How were standardized assessment instruments selected? 30 Areas of Assessment (Instruments) BASELINE ONLY (used for assessment but not to measure change over time) • DEPRESSIVE SYMPTOMS (Center for Epidemiologic Studies – CES-D) • ALCOHOL/DRUG PROBLEM SCREENING (CAGE-AID) • DOMESTIC VIOLENCE SCREENING (ACS SCREEN) • READINESS FOR CHANGE (Readiness to Change Index – REDI) • PARENTING ATTITUDES (Adult-Adolescent Parenting Inventory – AAPI-2) • SOCIAL SUPPORT (Support Functions Scale- SFS) • FAMILY RESOURCES (Family Resource Scale – FRS) • FAMILY FUNCTIONING (Family Functioning Style ScaleFFSS) • PARENTING SRESS (Parenting Stress Index-Short Form – PSI-SF) • HOME SAFETY-STABILITY (Developed for FCC) 31 NYC - FCC Logic Model Inputs ACS Funding Intermediate Outputs Final Intake/Outreach/Engagement Outputs Emergency Assistance (initial & ongoing) Trained Staff & Leadership Teams Implementation Planning Outcome Driven Service Plans (SMART goals) (6-8 weeks) Change Focused Intervention -Minimum 1 hour per week change focused intervention -Advocacy/service facilitation Evaluation of Change/ Case Closure(90 days post Plan) Long-Term Outcomes Increase Protective Factors Comprehensive Family Assessment (4-6 weeks) Eligibility Criteria and Referral Procedures Short-Term/ Intermediate Outcomes 1,536 Target Families • Parenting Attitudes • Family Strengths/Functioning • Social Support • Family Resources • Home Safety & Stability Decrease Risk Factors • • • • Parental Stress Caregiver Risks/Needs (FASP) Child Risks/Needs (FASP) Family Risks/Needs (FASP) Increase child Safety (Prevent Child Maltreatment) Achieve Permanency (Prevent Placement) 32 Developing the Plan to Build Competency 33 Overall Objectives for Building Competency for NYC Family Connections’ Practice • To build knowledge and skills related to the core components of Family Connections. • To practice working with “sample” families starting at Intake and ending at Case Closure. 34 Overview of Initial Training and Coaching Plan (N=42) (N=122) Initial Orientation April 9 & 12 (N=103) Supervisors and Directors Training April 22-23 & 29-30 Phase 1 Core Training (intake and engagement) May 1 or May 3 (N=103) (N=104) Phase 2 Core Training Phase 3 Core Training (family assessment/service planning) (change focused intervention; evaluating change; closure decision making) May 20-21 or May 30-31 July 9 & 10 or July 11 & 12 _________________________________________________________________________ April May June July Practicum 1 Practicum 2 Practice Introducing FC to Current Families 5/6 – 5/29 Coaching As needed Practice conducting comprehensive family assessment including use of instruments; practice developing a FC Case plan. Onsite Coaching June Continued Practicum Coaching with Supervisors – ongoing on at least a monthly basis 35 Coaching is unlocking a person’s potential to maximize their own performance. It is helping them to learn rather than teaching them ” - John Whitmore, 2002 36 Initial Implementation (Current Stage) Support ongoing implementation with fidelity, provide additional training and technical assistance as directed by fidelity assessments SAMPLE AREAS of focus • Provide monthly consultation, coaching, technical assistance • Revise self-assessment fidelity assessment instruments after initial fidelity assessment • Review agency self-assessment fidelity assessment instruments (every six months) • Conduct on-site fidelity reviews (2 per year) • Tailor ongoing technical assistance/training based on results of fidelity reviews Sample Fidelity Assessment Tool 38 Full Implementation What we have to look forward to • Learning is integrated – *The project becomes the practice • • • • • • • • Local capacity is built for ongoing training and coaching Staff feel confident in using the practice with every family Supervisors continually support case planners Stakeholders adapted to practice Procedures/processes are routine Practice change is observable (effective) Practice change is now the standard Implementation drivers are sustained 39 Implementation Drivers The core components that will increase the effectiveness of implementation 40 From Stages to Drivers • Drivers are implementation activities/components that support successful implementation. 41 Implementation Drivers Performance Assessment (Fidelity) Systems Intervention Coaching (and Supervision) Facilitative Administration Training Staff Selection Integrated & Compensatory Decision Support Data System LEADERSHIP © Fixsen & Blase, 2008 42 Implementation Drivers Competency Drivers Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention with fidelity and benefits to consumers. Competency Drivers include: Selection, Training, & Coaching leading to performance that meets fidelity Performance Assessment (Fidelity) Coaching and Supervision Training Staff Selection Integrated & Compensatory 43 Organization Drivers Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective services. Implementation Drivers Performance Assessment (Fidelity) Systems Intervention Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention *Performance assessment still included because all of these influence how well the practice is implemented Facilitative Administration Integrated & Compensatory Decision Support Data System 44 NYC FCC Organizational Assessment – Overview and Purpose • Purpose of initial NYC FCC Assessment: 1. 2. Assess the READINESS of agencies and staff across participating agencies to engage in the implementation of FC; and, Examine individual and organizational level factors that may SUPPORT or IMPEDE the successful implementation of FC. • Results are being used to inform the FCC implementation leadership team and individual agency teams – To respond to obvious worries or barriers for making the change – To ensure everything is in place to support this change in practice » Using the results is an example of the decision support data system driver 45 Work History from Organizational Assessment Work History of NYC FCC Respondents M SD Min Max Current Position 3.82 4.46 0.25 24.00 With Agency 6.23 7.27 0.33 36.00 Field of Child Welfare 11.75 8.26 0.0 35.17 46 Sample Philosophical Principles and Past Practice Extent to Which FC Philosophical Principles Drive Current Practice Helping Alliance Strengths-Based Empowerment Approach Cultural Differences 60.0% 46.3% 47.9% 50.0% 40.4% 40.0% 41.5% 55.8% 47.4% 43.2% 32.6% 30.0% 20.0% 10.0% 1.1% 1.1% 1.1% 1.1% 10.5% 10.6% 10.5% 7.4% 1.1% 1.1% 0.0% Not at all Slight extent Moderate extent Great extent Very great extent 47 Building Organizational Implementation Drivers Focus on Systems Intervention 48 Definition • The systems intervention driver includes strategies for working with external systems to ensure the availability of the financial, organizational, and human resources required to support implementation of the initiative (Fixsen et al., 2009) • One element of implementing this driver is establishing FCC Community Advisory Committees 49 Purpose of the NYC FCC Community Advisory Committees • To advise on the implementation of FC programs as well as to develop support to sustain the program on a long-term basis if effectiveness is demonstrated. – Committee members assist in recruitment strategies to directly identify families, community needs, and resources for both clients and the program, as well as help to remedy barriers to program implementation. – The committee directly participates in activities to build external stakeholder support and linkages on behalf of families who receive services from the program (e.g., may advise on newsletters for families and/or community providers). 50 51 52 Leadership Drivers Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Implementation Drivers Integrated & Compensatory LEADERSHIP 53 Implementation Drivers Integrated and Compensatory • Not a linear process • Drivers overlap and interact with each other • Drivers may be more or less salient at different stages • Strength of one driver can help to compensate for weakness in another Performance Assessment (Fidelity) Coaching and Systems Intervention Supervision Facilitative Administration Training Staff Selection Integrated & Compensatory Decision Support Data System LEADERSHIP 54 Questions/ Reflections?