Beyond logical frameworks to program impact pathways CIIFAD M&E Workshop 5 November 2011 135 Emerson Hall Sunny S. Kim, MPH PhD candidate Division of Nutritional Sciences Role of research in evaluation • Evaluation usually focuses on an internal situation, such as collecting data about specific programs, with no intent to generalize the results to other settings and situations. Evaluation determines the merit, worth, or value of things. • Evaluation asks what should be, which makes reference to the standards and criteria of evaluations. • Program-related research usually conducted with to the intent to generalize the findings from a sample to a larger population. • Research asks why, which indicates a theoretical basis. Basic steps of evaluation • Identifying program goals and objectives • Mapping the program • Selecting an evaluation design – Sample size, control group, sampling frame • Selecting indicators • Determining data collection strategy • Developing data collection instruments • Analyzing the data • Communicating the results Program theory vs. logic model • Program theory explains why a program is expected to work, and a logic model illustrates a program theory. • Program logic model is a picture of how your organization does its work… it links the theoretical assumptions of the program and program activities/ processes and outcomes. Translating program theory… Program theory has also been referred to as: • • • • • program logic or intervention logic theory-based or theory-driven evaluation theory of change theory-of-action impact pathway analysis all ways of developing a causal modal, linking program inputs and activities to intended or observed outcomes Translating program theory… • Program theories are often captured in a series of “if-then” statements. • And between the “if” and “then,” there should be “how” and “why”, i.e. evidence or well-established connections (established theory, previous results, published research, feedback from participants). IF… THEN… IF… THEN… + CONTEXT … into logic models • a commonly-used tool for illustrating program theory. • a picture of describing the sequence of activities to bring about change and how these activities are linked to the results the program is expected to achieve. • often presented in a flow chart that illustrates the linkages between program components and outcomes: Inputs Resources, supplies, staff Activities Processes, techniques, tools, events, technology, actions Outputs Direct results of activities (size and/or scope) Outcomes Changes in attitudes, behaviors, knowledge, skills, status, or level of functioning Impact Organizational, population, community, and/or system level changes (improved conditions, increased capacity, and/or policy changes) Program Logic Model Template … into logic models • “Logic models come in as many sizes and shapes as the programs they represent.” [W.F. Kellogg Foundation, 2001, p.7] • No best logic model, only the model that fits your program best and provides the information you need in the format that is most helpful. Why use a logic model? • Design and planning helps to explain and illustrate program concepts and approach for key stakeholders (advocate and train); finds “gaps” in the theory or logic of a program and work to resolve them . • Implementation and management focuses attention on the most important connections between actions and results for monitoring, improved programming, and reporting. • Evaluation, communication, and marketing presents progress toward goals; provides a way to involve and engage stakeholders in the design, processes, and use of evaluation. Why use a logic model? • Donor requirements: Log frame (OECD/DAC, USAID, CIDA…) Results framework (USAID…) Goals Objectives Outputs Activities Resources Assumptions Means of verification Indicators Goal: Health status improved Strategic Objective: Use of key services and behaviors improved Intermediate Result 1: Access and availability of services and supplies increased Intermediate Result 2: Quality of services improved Intermediate Result 3: Demand for key services and behaviors increased Intermediate Result 4: Social and policy environment enabled Logic model example Intervention model of Programa de Nutrición Infantil (PNI) COMPONENTS INTERMEDIATE OUTCOMES Maternal and child health and nutrition Basic sanitation and sanitary education Strategies: Information Education Social communication Improved health and nutrition knowledge and practices Increased access to health services Strengthening of civil society Improved basic sanitation Small-scale economic activities Improved development plans of CBOs in health and sanitation FINAL OUTCOMES Improved health and nutritional status of children <3 years Improved health and nutritional status of pregnant women Results framework of Programa de Nutrición Infantil (PNI) GOAL: Reduced rates of child malnutrition Strategic Objective: Improved health and nutritional status of children <3 years and pregnant women Intermediate Result 1: Improved nutrition, health knowledge and practices of those attended in the target communities Intermediate Result 2: Increased access to health services of the target population Intermediate Result 3: Improved basic sanitation in the target communities Intermediate Result 4: Community-based organizations applied their improved knowledge on health and sanitation in the formulation of their development plans From boxes to arrows… • While “filling in” the logic models with the program components , we must not neglect the mechanisms and linkages… don’t forget about your program theory or impact pathways. Description of mothers workshops “The theme/issue is the training that you give through the participatory methdology to the mothers. If you just give the theory, it is the theme. The workshop is the practice. So if that day, for example, the mothers didn’t bring food to prepare their workshop, then the [health] promoter only gives the theory. They discuss, socialize, and receive only the theories in the small groups… Then the workshop returns another day to learn about doing. For example, the health themes, they make their workshop do dramatizations. The mothers do their dramatizations, it tells the priority, what happens and everything, and so they are going to learn.” [Regional level, ex-ADRA adviser] Impact pathway of mothers workshops Household food availability and access Husbands permit/ support women to participate CHAs convene pregnant women and mothers of children <3 years to participate in workshops Women see incentive to participate (e.g. food baskets or small economic activities) CHAs raise topic of child feeding (i.e., BF and CF; balanced diets; feeding sick children) and nutrition for pregnant women CHAs raise topic of health and care for children (i.e., diarrhea and infections; respiratory illness; early stimulation) and reproductive health and prenatal care CHA raise topic of hygiene and sanitation CHAs raise topic of self esteem and family violence Influencing factors – facilitators CHAs and mothers discuss ideas and experiences CHAs and mothers demonstrate recipes Use of local language and readily accessible foods CHAs and mothers discuss ideas and experiences CHAs and mothers discuss ideas and experiences CHAs and mothers discuss ideas and experiences CHAs and mothers perform dramatizations CHAs and mothers demonstrate practices CHAs and mothers perform dramatizations Mothers understand nutritional needs of children and pregnant women and know feeding and cooking practices Mothers understand child health, reproductive health and prenatal care issues and know appropriate practices ? Mothers understand issues of hygiene and sanitation and know appropriate practices Mothers understand issues of self esteem and violence and know how to address them Mothers recall nutritional needs and feeding and cooking practices Repeated reminders or internal/external stimulus (e.g. home visits, concern for child’s future) Mothers recall child health, reproductive health and prenatal care issues and appropriate practices Mothers implement feeding practices and recipes Quality health service access and availability Mothers implement care practices and seek health services Children consume adequate and appropriate foods Pregnant women consume adequate and appropriate foods Pregnant women receive appropriate care and health attention in a timely manner Children receive appropriate care and health attention in a timely manner Household access to clean water Mothers recall information on hygiene and sanitation Mothers implement hygiene and sanitation practices Mothers recall information on self esteem and violence Mothers implement practices related to self esteem and violence Children exposed to safe and clean home environments and less exposure to contamination Children exposed to safe and caring home environments Husbands are sensitized about domestic violence Nutritional status of pregnant women improved Child nutrition and growth improved Main references • Kellogg Foundation. (2004). Logic model development guide: Using logic models to bring together planning, evaluation, and action. Battle Creek, MI: W.K. Kellogg Foundation. • Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds), Evaluation models, 209-232. Boston, MA: Kluwer Academic Publishers. • Rogers, P.J. (2008). Using program theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1): 29-48. • Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: teaching and training guide. Madison, WI: University of Wisconsin-Extension.