Approaches to Evaluating and Measuring Outcomes in Integrated Care Key Issues for Consideration Dr Nick Goodwin CEO, International Foundation for Integrated Care Senior Associate, The King’s Fund Paper to: Health Quality and Safety Commission New Zealand Workshop: Towards Integrated Care in New Zealand Wellington, New Zealand, November 14th 2013 What is a ‘programme evaluation’? • A systematic method of collecting, analysing and using information to answer questions about projects, policies or programmes • In health care systems, they are particularly concerned with quality of care, patient safety, system efficiency and/or cost effectiveness • They also seek to examine whether programme goals are, of have been, appropriate and/or useful so can be used to change and adapt strategic directions • They tend to utilise both qualitative as well as quantitative methods • They can be ‘formative’ or ‘summative’ Typical components of a ‘programme evaluation’ • Assessment of the need for the programme – Needs assessment/gap analysis/population health planning • Assessment of the design and/or theory and logic of the programme in supporting its desired influence – Are the assumptions of the programme justified? • Assessment of how the programme is being implemented – Process evaluation - is it going to plan? • Assessment of the programmes outcome or impact; cost and efficiency – Outcome level, outcome change, programme’s effect Rossi, Lipsey and Freeman (2004) Evaluation: a systems approach, Sage Understanding what to evaluate in an integrated care programme What are you evaluating – some key questions • Who and what is the programme seeking to influence? Need to clarify aim and design of the integrated care intervention by looking at the needs of patients/users • What is the timeframe over which outcomes are expected to be achieved? Given this timeframe, which categories of outcomes have the potential to be improved? • Is there sufficient opportunity in a given population to achieve this targeted improvement in outcomes? • How can you measure the impact? How can you ensure attribution? What are you evaluating – some key questions Before developing questions and/or survey instruments to examine the experience and impact of integrated care from a person’s perspective, there is a need to understand four things: the programme theory of change – what are the assumptions that lie behind the programme (why?) the (set of) problems to be addressed (where and who?) the (set of) interventions best suited to address the problem (what and who?) the strategy best suited to develop, implement, and evaluate the (set of) interventions (how, when and who?) What are you evaluating – some key questions For integrated care to be successful, it needs to execute the following three functions: accurate identification of individuals within target population (e.g. reliable predictive modelling, health risk assessment, medication list and/or laboratory values from EMRs); individuals must be enrolled and actively participate in the program for a meaningful period of time (e.g. readiness to change, motivational interviewing, incentives); the program must include a set of interventions that modify or close deficits in participant and provider behaviour (e.g. tailoring to needs). Key Points to Consider Baseline data Define a comparison group Define nature and structure of integrated care being implemented Include measures of the professionals’ perspective where care is delivered through multidisciplinary teams Identify what good looks like from a patients’ perspective and evaluate this through user feedback Include analysis of utilisation and costs of care Experiences, care outcomes, utilisation & costs Understanding what to measure in an integrated care programme Measurement Types - 1 Care Outcome Measures Patient outcomes – e.g. mortality, morbidity, functional status, quality of life Cost and utilisation outcomes – e.g. hospital admissions, bed days, LOS, nursing home placements Care Process Measures Occurrence of recommended care activities – e.g. presence of a care plan; patient follow-up – often processes that are set out in best practice guidance The measurement tools we have on outcomes and processes are mostly disease-specific - for people with multiple needs, the process of care is less well understood Examples of outcome measures Hospital utilisation – Emergency admissions; hospital readmissions; lengths of stay; number of bed days etc … – Disease-specific hospital admissions etc … ‘Social care’ utilisation – Levels of home care support packages – Rates of long-term nursing home/residential home stays Mortality and disease-specific mortality Short-term clinical outcomes – e.g., glycated haemoglobin levels for diabetic patients Functional status – e.g., for CHF patients Quality of life – e.g., functional dependence Other patient outcomes – e.g., missed school days for children due to illness; experiences with system Treatment and service adherence – e.g., remaining in contact with services for mentally ill patients Measurement Types - 2 Measures of care co-ordination Information exchanges and transfers Relational co-ordination between organisations and professionals – levels of awareness/interaction among participants – is there a common understanding of care activities and goals – shared culture and mission? Co-ordination Mechanism Measures Approaches Direct observation Interviews and staff surveys (self-report) Medical record audits on information transfer Measures of inter-professional collaboration within teams and organisations Examples: ITMA - Integrated Team Monitoring and Assessment Tool – see http://www.readiness-tools.com/tool-full.aspx?toolguid=0d6382ad-f0174623-8d10-93f2f314e346 POET – Partnership Outcomes Evaluation Toolkit – see www.dhcarenetworks.org.uk/asset.cfm?aid=1479 Care Co-ordination Measures Atlas McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf and http://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools Domains for measuring care co-ordination Co-ordination activity: Service delivery approaches: Establish accountability/negotiate responsibility Communication – informational and inter-personal Facilitate transitions – e.g. across settings or as coordination needs change Assess multiple needs and goals Pro-active care planning Monitor, follow-up, review Support self-management Link or refer to community resources Align resources to meet individual or community needs Care management Medicines management Healthcare at Home Multi-disciplinary teams ICT-enabled integrated care (e.g. telehealth) Perspectives: Family/patient Professional System/organisation Measurement Types - 3 Patient, carer, family-reported perceptions PROMs • • do you feel better? have your symptoms improved? PREMs • • recommend to a friend? how satisfied are you? * PACIC patient assessment of chronic illness care http://www.improvingchroniccare.org/downloads/2004pacic.doc.pdf * Care Transition Measures (CTM) – patient assessment of discharge from hospital http://www.caretransitions.org/documents/CTM-15.pdf The more ‘robust’ measures are those that ask about specifics of care co-ordination rather than generalities, and which link data sets together to understand association between responses. Two key things: measures that are specific to a particular group of people; measures that translate to actionable service improvements Patient Reported Outcome Measures in England Since 2009, PROMs have been collected for four elective procedures: hip surgery, knee surgery, hernia repair and varicose veins – these costs the English NHS £800m/year Cataract surgery was to be included, but concerns on methods of data collection 250,000 patients a year invited to complete survey – all NHS providers (100%) collect data PROMS being extended to cover: anxiety and depression, cancer care, asthma, COPD, diabetes, epilepsy, heart failure, stroke It’s a careful process – identify the right PROMs instrument; pilot before roll-out; implement data collection; evaluate programme Uses EQ-5D – part 1 about mobility, selfcare, usual activities, pain/discomfort, anxiety/depression; part 2 gives overall assessment of health on 0-100 scale NOTE: Not sensitive to looking at people with multiple conditions/needs Some Conclusions on Measuring Integration Many different tools available: Need to define the client group Need to understand the goal in terms of outcomes to patients and service users Need to create ‘measurable’ outcomes and experiences Measures need to mean something – i.e. that actions can follow Patients and users tend to understand the term ‘care co-ordination’ or ‘continuity of care’ – e.g. to what extent they feel that care is co-ordinated around their needs Baseline on measures required on which to base progress over time Link measures to other data – e.g. on clinical outcomes, utilisation, costs Where possible, benchmark performance with others or investigate with a matched ‘control’ Use data in ‘real time’ to monitor progress and drive performance Case Example Integrated Care Metrics NHS London (2012) Activity metrics currently used in IC systems in London, across all settings of care Metric Definition Emergency admissions (No.) Number of unplanned admissions for ACS conditions that should not usually require hospital admission Number of emergency admissions for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients) Case file audit results for reviews of avoidable ED admissions Emergency bed days (No.) Number / total number of emergency hospital bed days for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients) Emergency bed days associated with multiple acute hospital admissions SUS/ HA data ONEL, NWL, ELC A and E attendances (No.) Number / Total number of A and E attendances for the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients) SUS data NWL, ONEL, NCL, KHP, ELC Acute re-admissions (No/ %) All emergency re-admissions that occur within 30 days of discharge for IC cohort (over 65/75/ top ICD 10 codes/ community ward patients) SUS / HA data NWL, ONEL, KHP Length of stay (No.) Data source SUS / HA data Difference between the discharge time and the admittance time for both Dr Foster/ SUS / HA elective and non elective episodes divided by the total number of spells for IC data cohort Average Length of Stay for patients in the IC cohort (over 65/75/ top ICD 10 codes/ community ward patients) ICS NWL, ONEL, NCL, ONEL,ELC, KHP NWL, ONEL, NCL, ELC Permanent admissions to residential and nursing care homes (No./ %) Number of patients in long term care homes as a proportion of the total number of patients in the pilot (%) Number of new permanent admissions to residential and nursing care homes, 65+/ -65 Case file audit results for reviews of recent care home admissions Referral Assessment Provision (RAP) stats KHP, NWL, Greenwich, NCL, ONEL Number of people completing re-ablement (No.) Proportion of service users independent following re-ablement Readmission within 1 year for patients who have completed re-ablement Referral Assessment Provision (RAP) stats Greenwich RiO NWL Community nursing hours per person (No.) Total number of community nursing visits or units delivered divided by number of patients in pilot People supported at home with low/med/high care packages (No.) Number of people 65+supported at home with low/med/high care packages Referral Assessment Provision (RAP) stats KHP, Greenwich Urgent GP response DH survey (Q14-17 seen on same day within primary care 'how convenient' score) QOF KHP Process metrics currently used by IC systems in London Metric Definitions Source ICS Number of health assessments completed (No.) Number of assessments complete versus target Outcomes of health assessments Emergency admissions of patients who had health assessments Local ICS collection KHP Number of people case managed (No.) Numbers of case managed patients Caseload, analysed by length of stay, source of referral and state of care plan % currently case managed patients with no ED contact ‘Referral bounce’ for case managed patients Local ICS collection KHP Number of care plans completed (No.) Numbers of patients with personalised care plans Number of people in cohort on a care plan as a proportion of total number of people that should be on a care plan Local ICS collection ONEL , NWL Adherence to care plan (No.) Number of patients with one or more delay or incomplete actions as a proportion of the total number of patients in the pilot (%) Local ICS collection NWL MDT operation (No.) Numbers of case managed patients discussed at MDT for whom follow up actions took place Local ICS collection KHP Attendance at MDT meetings (No.) Number of meetings scheduled / held / fully attended Number of clinicians attending monthly MDT case conferences and quarterly review meetings as proportion of total number expected at these meetings (%) Local audit KHP Frequency of MDT meetings (No.) Virtual Ward: number of monthly MDT meetings held Local ICS collection ELC Degree of joint working Attendance rates at joint meetings Exception reporting – local ICS collection ONEL Outcome metrics currently used by IC systems in London Metric Control measures Hard outcomes Definition Source Patients with T1/T2 DM with HbA1c less than or equal to 10 QOF % of patients with blood pressure targets achieved as per NICE guidance Percentage of patients with cholesterol less than or equal to 5 BMI less than or equal to 30 proportion of patients receiving medication review who should Speed of referral for recognised foot complications, QOF Amputation rate below the ankle, Falls rate among the frail elderly, Number of fractures ICS NWL NWL Patient and staff experience measures used by IC systems in London Metric Definition Source ICS PROMS Measure of quality from the patient perspective covering four clinical procedures: hip replacements, knee replacements, hernia, varicose veins NHS Information Centre NWL, ONEL PREMS Measure of patients experience based on range of different possible metrics: patient recommendation, overall satisfaction, complaints, patient confidence Patient surveys and questionnaires, focus groups, complaints data, one on one patient interviews NWL , ONEL Patient satisfaction surveys Results of ICS service user survey Percentage of people with LTCs who feel supported to manage their condition The proportion of people who use services who feel safe DH GP survey 032 KHP, ONEL LTC6 survey 1. Did you discuss what was most important for you in managing your own health? 2. Were you involved as much as you wanted to be in decisions about your care or treatment? 3. How would you describe the amount of information you received to help you to manage your health? 4. Have you had enough support from your health and social care team to help you to manage your health? 5. Do you think the support and care you receive is joined up and working for you? Local health observatory 6. How confident are you that you can manage your own health? Staff satisfaction Staff survey to determine satisfaction of IC pilot, to determine number of staff responding very satisfied and satisfied to survey as proportion of total number of staff surveyed Local ICS collection NWL Contact Dr Nick Goodwin CEO, International Foundation for Integrated Care firstname.lastname@example.org www.integratedcarefoundation.org @goodwin_nick @IFICinfo Appendix 1: From a measure to an indicator that can be used for quality improvement In developing ‘indicators’ to judge comparative performance, the following criteria are important: Statistical validity Accurate – measures what is says it measures! Reliable – can be tracked over time Consistent – data collection robust and reproducable Avoids bias Data considerations Data source – sample or full population; existing or new data source? Unit of assessment – country, region, locality, practice etc Client groups covered – what is the targeted population? account for bias/case mix? Significance – smaller groups, reduced statistical power Patient reported data is indirect – how cross-compare to other data (multi-methods) Coverage – health, health & social care, health, social care and housing …. Face validity Meaningful to public; clinically credible; Potential to support quality improvement; Cost and value for money Appendix 2: Key Resources on Care Co-ordination Measures McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf ttp://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools King et al - 1995/2004 - Measures in the process of care – MPOC http://www.canchild.ca/en/measures/mpoc56_mpoc20.asp Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract. 1997 Jul;45(1):64-74. Safran DG, Kosinski M, Tarlov AR, et al. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care. 1998 May;36(5):728-39. Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer experiences with primary care. Pediatrics. 2000 Apr;105(4 Pt 2):998-1003. Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract. 1998;46(3):216-26. Grimmer K, Moss J. The development, validity and application of a new instrument to assess the quality of discharge planning activities from the community perspective. Int J Qual Health Care. 2001 Apr;13(2):109-16 McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of health care. Int J Qual Health Care. 2003 Aug;15(4):309-18. Radwin L, Alster K, Rubin KM. Development and testing of the Oncology Patients' Perceptions of the Quality of Nursing Care Scale. Oncol Nurs Forum. 2003 Mar- Apr;30(2):283-90. Billings J, Coxon K, Alaszewski A. Empirical research methodology for ‘Procare’ version 3. University of Kent at Canterbury: Centre for Health Services Studies; 2003. Available from: http://www.kent.ac.uk/chss/docs/procare_version3.pdf. Morita T, Hirai K, Sakaguchi Y, Maeyama E, Tsuneto S, Shima Y. Measuring the quality of structure and process in end-of-life care from the bereaved family perspective. J Pain Symptom Manage. 2004 Jun;27(6):492-501. Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient's perspective: the care transitions measure. Med Care. 2005 Mar;43(3):246-55. (also CTM website) Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC).Med Care. 2005 May;43(5):436-44 Drewes et al (2012) PCIC+ at http://www.ijic.org/index.php/ijic/article/view/862/1896 Haggerty J, Roberge D, Freeman G, Beaulieu C, Breton M. Review. When patients encounter several providers: validation of a generic measure of continuity of care. Annals of Family Medicine (In press) and Breton et al at http://www.ijic.org/index.php/ijic/article/view/682/1534 Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): Conceptualizing and Measuring Activation in patients and consumers. Health Services Research 2004: 39:1005-1026. Powell, R. Powell, H. Baker, L. & Greco, M. (2009) Patient Partnership in Care: A new instrument for measuring patient–professional partnership in the treatment of long-term conditions. Journal of Management & Marketing in Healthcare. Vol. 2 No. 4. PP 325–342. Long Term Conditions 6 (LTC 6) (part of the QIPP programme) Department of Health and also Living with your Long Term Condition (LWYLTC) Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al. Measuring Patients' Experiences with Individual Primary Care Physicians Results of a Statewide Demonstration Project. Journal of General Internal Medicine. 2006;21(1):13-21. Adair CE, McDougall GM, Mitton CR, Joyce AS, Wild TC, Gordon A, et al. Continuity of Care and Health Outcomes Among Persons With Severe Mental Illness. Psychiatric Services. 2005;56(9):1061-9. King M, Jones L, Nazareth I. Concern and continuity in the care of cancer patients and their carers: A multi-method approach to enlightened management. London: National Coordinating Centre for NHS Service Delivery and Organisation 2006. Dolovich LR, Nair KM, Ciliska DK, Lee HN, Birch S, Gafni A, et al. The Diabetes Continuity of Care Scale: the development and initial evaluation of a questionnaire that measures continuity of care from the patient perspective. Health Soc.Care Community. 2004;12(6):475-87 Kowalyk KM, Hadjistavropoulos HD, Biem HJ. Measuring Continuity of Care for Cardiac Patients: Development of a Patient Self-report Questionnaire. Canadian Journal of Cardiology. 2004;20(2):205-12. Hadjistavropoulos H, Biem H, Sharpe D, Bourgault-Fagnou M, Janzen J. Patient Perceptions of Hospital Discharge: Reliability and Validity of a Patient Continuity of Care Questionnaire. International Journal for Quality in Health Care : Journal of the International Society for Quality in Health Care / ISQua. 2008;20(5):314-23. Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based outpatient clinics: performance measures based on patient perceptions of care. Med Care. 2002;40(7):578-86.