Namibia looking forward: Effectiveness of capacity development support to local partners Stakeholder data validation workshop July 2, 2012 Windhoek Why we are here today • To share Pact Namibia’s internal evaluation findings with stakeholders • To get stakeholder validation of the evaluation recommendations Agenda Time Activity 9:30–10:45 Presentations 10:45–11:00 Tea break 11:00–11:30 Q&A 11:30–noon Small group work: Discuss findings & review recommendations Group 1. Capacity development impact Group 2. Graduation to direct USAID funding Group 3. Lessons learned for capacity development organizations Noon–12:45 Report back 12:45–1:00 Conclusions & next steps 1:00–2:00 Lunch Background • USAID Namibia began scaling up HIV programs in 2003. • In 2007 USAID awarded Pact a 5-year project which included grants, organizational and technical assistance to CSOs, predominantly in areas of: • HIV/AIDS prevention • Care and support to orphans, HIV-vulnerable children and home-based clients • Strengthening capacity of the Ministry of Gender Equality and Child Welfare Background • In 2010 – award changed to focus on two objectives: 1. Strengthening capacity of the Ministry of Gender Equality and Child Welfare to effectively coordinate activities relating to gender equality and orphans and vulnerable children 2. Strengthening six civil-society organizations to be eligible for U.S. direct funding while maintaining high standards of existing services Tools to strengthen civil society • Organizational Development (OD) Roadmaps measures CSO partner organizational systems and structures objectively • Basis for identifying organizational efficiency gaps and prioritizing interventions • Progression on a scale from 1 (nascent) to 5 (mature) • Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacitybuilding activities Tools to strengthen civil society • Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacitybuilding activities • Organizational Development (OD) Roadmaps • 10 capacity areas: Monitoring & evaluation Organizational sustainability Grants & compliance Purpose & HR management planning Governanc Networking e Financial management Operations Projects & management services Organizational Performance Index (OPI) Scored on a scale of 1 to 4 on each of the following dimensions: • Effectiveness. Achieving outcomes, meeting standards • Efficiency. Delivering services, increasing reach • Relevance. Engaging target populations, promoting learning • Sustainability. Mobilizing resources, increasing legitimacy Why do this evaluation? 1. Assess the sustainability of graduated partners’ capacity development 2. Review the quality and quantity of services offered to beneficiaries of former and current CSO partners (partially done) 3. Assess the effectiveness of the Pact Namibia capacity building assistance to the partners that have graduated or plan to graduate 4. Draw lessons for other Pact programs that are facing similar challenges of graduating partners Evaluation questions • To what extent was Pact Namibia’s capacity development support effective? • To what extent were partners ready for and able to effectively manage direct funding? • To what extent did service quality and coverage change over the last two years? Process 3 phases: 1. Designed terms of reference and data collection tools (virtual group) 2. Collected data (Pact staff from 5 country offices, for neutrality) 3. Analyzed data, drafted and disseminated report (current group of Pact staff) Data collection tools 1. Partner survey (Mobenzi) • 17 current & former partner organizations interviewed • Quantitative & qualitative 2. Organizational Performance Index (OPI) • 12 of 17 partners 3. Pact staff survey (Mobenzi) • 4 core Pact Staff from each technical area 4. USAID tool 5. Historical OD Roadmap scores Data analysis • Quantitative data: • Merged datasets and did selected frequencies with statistics and cross-tabulations using SPSS and Excel. • Qualitative data: thematic analysis. Partner breakdown OD Roadmap support, graduated to direct funding OD Roadmap support, not graduated No OD Roadmap support, graduated No OD Total Roadmap support, not graduated Number of orgs 3 (18%) 3 (18%) 2 (12%) 9 (53%) 17 Number of partner respondents 11 (23%) 12 (25%) 3 (6%) 22 (46%) 48 Study limitations Sample size • Only 17 organizations • Limited number of respondents in some organizations; some interviewed were previous staff Time • Could not conduct survey of beneficiaries Presentation of the findings • Capacity development impact • Graduation to direct USAID Funding • Lessons learned for capacity development organizations • Preliminary recommendations Capacity development impact Key areas of analysis 1. How did length of support influence partners’ OPI scores and perceptions of usefulness of support? 2. How do Pact and partner perceptions of support compare? 3. How do OD Roadmap supported partners differ from other partners? 4. Which areas of Pact support did partners rate as most useful and of highest quality? 5. What were the most significant changes resulting from Pact’s support? 1. Partners scored longer support less useful Usefulness rating 5.0 4.5 4.0 3.5 Monitoring & evaluat Strategic Financial Program thematic 3.0 1-3 years of support 3+ years of support 2. But Pact scored longer support more useful 4.5 4.0 3.5 3.0 1-3 years of support 5.0 M&E Strategic Financial Program thematic 3+ years of support Partners score longer support less useful Pact rating Partner rating 5.0 4.5 4.0 3.5 Strategic M&E Program thematic Financial 3.0 1-3 years of support 3+ years of support Pact scores longer support more useful (except financial) % of partners scoring over 3 3. Longer support – higher OPI scores 80% 60% 40% Program thematic Monitoring & evaluat Financial Strategic 20% 0% 1-3 years of support 3+ years of support OPI score above 3 4. OD Roadmap partners scored higher on OPI 100% 67% 50% 17% 0% OD Roadmap Non-OD Roadmap 1a. But non-graduates more keen on financial management % of all partner staff 60% 50% 40% 30% Non-graduated partners 20% 10% 0% Technical Support Financial management support M and E reporting support Advocacy Support Strategic planning support HR Management Other 5. Non-OD Roadmap partners rated Pact support more useful Usefulness rating 4.50 4.25 Non-OD Roadmap 4.00 3.75 OD Roadmap 3.50 Strategic Support Financial Support Program Thematic M&E 6. Partners rated M&E support highest 4.50 Usefulness rating 4.25 4.00 Quality rating 3.75 3.50 Program Technical Financial Planning Strategic Planning M&E 7. Most significant changes M&E and Financial management the “cornerstones of programs” (partner, Pact and USAID responses) • M&E • Timely reporting • Capturing the right and required data • Use data for decision making • Finance • Proper management of organization finance & USAID funds • Complete and accurate finance reports • Proper budget management 7. Most significant changes • Moved from no system to very good system in all areas (partner responses) • Buy-in from organizations (Pact responses) • How do we sustain the change? • Experience handover and staff retention • Resources Summing up • More years of support = lower usefulness rating, but higher OPI scores • Pact and partners see usefulness differently over the duration of support • OD Roadmap support = lower usefulness rating, but higher OPI scores • M&E support rated most useful and of highest quality • Most significant change in M&E and financial management Graduation to direct USAID funding Key areas of analysis 1. Capacity-building support that partners would like more of before graduation 2. Graduate vs. non-graduate performance on OD Roadmap and OPI 3. Prerequisites for graduation 4. Key capacity-building support for graduated partners 1. Partners want more technical support before graduation % of all partner staff 60% 50% 40% 30% 20% 10% 0% Technical Support Financial management support M and E reporting support Advocacy Support Strategic planning support HR Management Other 1b. And graduates far more interested in technical support % of all partner staff 60% 50% Graduated partners 40% 30% Non-graduated partners 20% 10% 0% Technical Support Financial management support M and E reporting support Advocacy Support Strategic planning support HR Management Other 2a. Graduates scored higher on OD Roadmap 4 Graduates 3.5 3 Nongraduates 2.5 2 1.5 1 0 1 2 3 4 5 6 7 8 9 10 11 2b. But OPI not different 4 3.5 3 2.5 2 1.5 1 Graduates Non-graduates 0 1 2 3 4 5 3. Graduation prerequisites from Pact and partners 1. 2. 3. 4. Strategic plan Governance Strong financial systems and internal controls Administrative systems, including human resource management 5. Skilled staff and technical capacity 6. Strong monitoring and evaluation 4. Key post-graduation capacity development Priority • Resource mobilization support • Technical/Program support As needed • Monitoring and evaluation • Database development • Evaluations • Routine data quality assessments • Financial management Summing up 1. Graduated partners would have liked more technical support before graduation 2. Graduated partners averaged 3.4 on Roadmap, up from 2.5 for non-graduated partners (no difference on OPI) 3. Six graduation prerequisites: strategic plan, governance, finance, admin, technical capacity, M&E 4. Key post-graduation needs: technical and resource mobilization support Lessons learned for capacity development organizations Key areas of analysis 1. Retention of systems after capacity building, and impact of who decided to engage in it 2. OPI scores and trends across partner subgroups 3. Different capacity needs identified by partners and donors 4. Challenges and advantages of graduating to direct funding 1a. OD Roadmap-supported partners more likely to retain systems % of orgs still using system OD Roadmap Support 100% 80% 60% Non-OD Roadmap Support 40% 20% Strategic Planning Financial Management Program Thematic Area Monitoring & Evaluation 1b. Retention likelier if system is partnerdriven Who decides to acquire system % retaining system N Partner alone 95% 60 Both partner and Pact 83% 87 Pact alone 75% 24 1c. Retention likelier if system is partnerdriven Strategic Monitoring & evaluation % retaining system 100% 80% Financial 60% Program thematic 40% 20% Partner's Decision Both Decision Pact's Decision 2. OPI scores • Average OPI scores show lower resource mobilization but high social capital • Graduated partners generally score higher • Current partners score low on resource mobilization and service delivery 2a. Average OPI shows lower resource mobilization, high social capital 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 Average OPI Effectiveness Results Efficiency Standards Delivery Reach Relevance Target Sustainability Learning Resources Social Capital 2b. Graduates need to focus on results and external evaluation of 4.0 standards OD Roadmap graduates 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 Average OPI Effectiveness Results Efficiency Standards Delivery Reach Relevance Target Sustainability Learning Resources Social Capital 2b. Current partners scored lower on resource mobilization and service delivery 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 Non-graduated OD Roadmap partners Average OPI Effectiveness Results Efficiency Standards Delivery Reach Relevance Target Sustainability Learning Resources Social Capital 3a. Different stakeholders, different top-ranked capacity needs OD Roadmap partners USAID 1. Monitoring & evaluation 1. Resource mobilization 2. Technical support 2. Financial management 3. Financial management 3. Administration 4. Program development 4. Monitoring & evaluation Non-OD Roadmap partners Pact 1. Technical support 2. Financial management 2. Financial management 3. Institutional support 3. Advocacy 4. Technical support 4. Monitoring & evaluation 1. Resource mobilization 3b. Graduated partners & USAID: contrasting expectations Partners expect: USAID expects: 1. Same relationship/treatment with USAID as partners had with Pact 1. Strong organizational capacity and governance systems, ability to work independently and maintain capacity even after staff turnover 2. Continued technical assistance, but often lack adequate budget 3. Increased USAID capacitybuilding support after graduation 2. USAID support to partners, but also partner ability to address own needs 3. Performance like long-term international USAID partners, responsive and accountable 4. Graduates found both challenges & benefits Challenges Benefits 1. Increased accountability in line with U.S. Government rules & regulations 1. Shorter reporting channels give more time to prepare reports 2. Less frequent capacitybuilding support 2. Improved interaction with USAID & understanding of USAID programs 3. Would like clearer guidance from USAID on budgets, funding & programs 4. Would like more regular feedback on financial reports 3. Less cumbersome financial reporting requirements 4. International exposure & recognition 5. Good working relations Summing up 1. Current partners most likely to retain M&E, financial and technical systems 2. Sustainability of capacity building depends on whose decision it was to engage in capacity development 3. On average partners tend to score lower in resource mobilization on OPI scoring 4. Partners still need to develop capacity, but are struggling to understand how to access support under direct funding. Preliminary recommendations Preliminary recommendations • Capacity development impact • Graduation to direct USAID funding • Lessons learned for capacity development (CD) organizations • General Capacity development impact • Capacity development support should be provided for three or more years (more research is needed). • Capacity development organizations need to keep their support “fresh” for organizations served for longer periods. • OD roadmap support is an important approach for preparing effective partners and should continue to be used. • Capacity development support should increase focus on knowledge transfer and resource mobilization for increased sustainability. Graduation to direct USAID funding • The OD Roadmap should be considered as the tool to measure preparedness for an organization’s readiness for graduation. • The prerequisites for transition to direct funding should include: • • • • • • • • Strong strategic planning capacity Governance structures Strong financial systems and internal controls Strong administration (including HR) systems Skilled staff and technical capacity Strong M&E An ability to mobilize resources An ability to advocate with donors to meet (internal and external) constituent needs. Lesson learned for CD organizations • CD organizations should ensure ownership of interventions for maximum impact in the area of system retention. • CD organizations need to work with partners to diversify their funding base. • CD service providers need to help partners to remain relevant and vibrant by planning to incorporate internal capacity development support on an on-going basis. • Increased communication on expectations will improve the transition to direct funding for both partners and USAID. General • Pact should consider expanding this dataset to include more Namibian respondents and more international graduates (33 worldwide) and partners in order to make the findings more robust. • Pact should assess the impact of the capacity development approach (and the transition to direct funding) at the beneficiary level. Thank you!