Dr Ian Goldman Natioanal Evaluation System Presentation

Report
The Presidency
Department of Performance Monitoring and Evaluation
National Evaluation System
Dr Ian Goldman,
Head: Evaluation and Research
PSPPD PSC
26 August 2014
Overall situation with evaluations
 39 evaluations completed, or underway, of
which 37 are NEP evaluations.
Improvement
plans being
implemented
8
Served Approved Research TORs
Preparatio Stuck
at
reports
underway approved n stage
Cabinet
2
11
23
0
2
2
Cost benefit - R50 billion of programmes evaluated
(over MTEF). 10% improvement = R5 billion, annual
cost of evaluation system R50 million, so very high
return on investment.
The Presidency: Department of Performance Monitoring and Evaluation
2
Nos of departments involved
 15/46 national
departments now
involved in evaluations
 Nos of national +
provincial departments
involved in evaluations
risen from 13% to 23%
The Presidency: Department of Performance Monitoring and Evaluation
3
NEP 2012/13
Dept
dti
DBE
Health
DRDLR
DRDLR
Completing
Underway
Title of evaluation
Implementation/ design evaluation of the
Business Process Services Programme
Impact Evaluation of Grade R
Implementation Evaluation of Nutrition
Programmes addressing Children Under
5
Implementation Evaluation of the Land
Reform Recapitalisation and
Development Programme
Implementation Evaluation of CRDP
About to start
Delays!
Status as 5 May
Final report approved. Report
on way to Cabinet
Final report approved. Report
been to Cabinet
Final report approved and
been to Cab Committee
Final report approved
been to Cab Committee
and
Final report approved and
been to Cab Committee
DHS
Implementation Evaluation of the
Evaluation underway. DHS
Integrated Residential Dev. Programme procurement has delayed
DHS
Implementation Evaluation of the Urban Evaluation underway. DHS
Settlements Development Grant (USDG) procurement has delayed
DBE
Impact Evaluation of the National School DBE requested to drop.
Nutrition Programme (NSNP)
Reallocated to 2014/15.
4
The Presidency: Department of Performance Monitoring and Evaluation
NEP 2013/14
Completing
by Sept
Underway
About to start
Delays!
Dept
Title of evaluation
Presidency
dti
Implementation Evaluation of Government’s Coordination Systems
Evaluation of Export Marketing Investment Assistance Incentive programme
(EMIAI)
Evaluation of Support Programme for Industrial Innovation (SPII)
Impact Evaluation of Technology and Human Resources for Industry
Programme (THRIP)
Evaluation of Military Veterans Economic Empowerment and Skills
Transferability and Recognition Programme.
Evaluation of National Advanced Manufacturing Technology Strategy
Impact Evaluation on Tax Compliance Cost of small businesses
Impact evaluation of the Community Works Programme (CWP)
Evaluation of the Land Restitution Programme
Impact Evaluation of CASP
Implementation Evaluation of MAFISA
Baseline for informal settlements targeted for upgrading
Evaluating interventions by DHS to facilitate access to the city.
Diagnostic of whether the provision of state-subsidised housing has
addressed asset poverty for households and local municipalities
Impact Evaluation of the Outcomes Approach
dti
dti
Military
Veterans
DST
SARS
COGTA
DRDLR
DAFF
DAFF
DHS
DHS
DHS
DPME
The Presidency: Department of Performance Monitoring and Evaluation
5
2014/15
Underway
TORs
developed
No TORs
yet
Delays!
Dept
Evaluation
DEA
DAFF/DRDLR
DAFF
Evaluation of the Effectiveness of Environmental Governance in the Mining
Sector (EEGM)
Design Evaluation of the Policy on Community Education and Training
Colleges (PCETC)
Impact Evaluation of the Social Housing Programme (SHP)
Evaluation of the Indigenous Knowledge Systems Policy (IKSP)
Diagnostic Evaluation/Programme Audit for Violence Against Women and
Children (AVAWC)
Diagnostic Review of Coordination of the Social Sector Expanded Public
Works Programme
Economic Evaluation of the incremental investment into the SAPS Forensic
Services
Impact Evaluation of the Ilima Letsema Programme and Irrigation Schemes
Impact evaluation of MAFISA (quantitative) – through 3ie
DAFF/DRDLR
DBE
DBE
Policy Evaluation of Small Farmer Support
Evaluation of the Funza-Lushaka Bursary Scheme
Impact evaluation of National School Nutrition Programme
DRDLR
Impact evaluation of Land Restitution Programme – through 3ie
DPME
DPME
Impact/implementation evaluation of the MPAT system
Implementation evaluation of the dept strategic planning and APP system
DHET
DHS
DST
DSD
DSD
SAPS
The Presidency: Department of Performance Monitoring and Evaluation
6
Evaluations selected for 2015/16











Impact/implementation evaluation of the evaluation system (DPME).
Implementation evaluation of citizen-based monitoring programme
(DPME).
Agricultural Extension Recovery Plan (DAFF).
Evaluation of the New School Curriculum (DBE).
Evaluation of the Asset Forfeiture Unit Sub-programme (NPA).
Diagnostic evaluation of the Non-Profit Organisations Regulatory
Framework and Legislation (DSD).
Implementation Evaluation of the National Drug Master Plan in
addressing all forms of Substance abuse (DSD).
Evaluation of the effectiveness of the South African Post-School
Education and Training (PSET) quality assurance regime (DHET).
Evaluation of Early Grade Reading in SA (DBE).
Evaluation of the Mining Charter
Evaluation of the service delivery improvement planning system
The Presidency: Department of Performance Monitoring and Evaluation
7
MTSF Outcome
All
evaluations
since 2009
Nos of
evaluations
in NEPs
1.
2.
3.
4.
Improved quality of basic education.
15
5
A long and healthy life for all South Africans.
3
1
All people in South Africa are and feel safe.
4
3
Decent employment through inclusive economic
14
9
growth.
5. A skilled and capable workforce to support an
9
3
inclusive growth path.
6. An efficient, competitive and responsive economic
1
0
infrastructure network.
7. Vibrant, equitable and sustainable rural
12
8
communities with food security for all.
8. Sustainable human settlements and improved
7
6
quality of household life.
9. A responsive, accountable, effective and efficient
3
2
local government system.
10. Environmental assets and natural resources that
5
1-2
are well protected and continually enhanced.
11. Create a better South Africa and contribute to a
0
0
better and safer Africa and World.
12. An efficient, effective and development oriented
10
5
public service and an empowered, fair and
inclusive citizenship
13. Social Protection
8
6
The Presidency: Department of Performance Monitoring3 and Evaluation
14. Social Cohesion
1
8
Programme being evaluated
Early Childhood Development (ECD)
Business Process Services Programme
Grade R (reception year of schooling)
Nutrition Programmes addressing under 5s
Land Recapit. and Dev Prog (RECAP)
Comprehensive Rural Dev Prog (CRDP)
Export Marketing Investment Assistance
Support Prog for Industrial Innovation
(SPII)
Land Restitution Programme
Government Coordination System
(clusters/MinMECs/Implementation
Forums)
Micro Agric Financial Institution (MAFISA)
Presidential Intervention in KSD
Outcome needed from evaluation
results
Close Major
Minor
No
changes changes changes
needed
needed
needed
X
X
X
X
X
X
X
X
X
X
Many
programmes
need major
changes
X
X
The Presidency: Department of Performance Monitoring and Evaluation
9
Progress with
National Evaluation System







3 National and Provincial Evaluation plans to date, last week selected
evaluations for 4th
39 evaluations completed, underway (11 completed, 2 public)
18 guidelines, evaluation standards, competences, trained >600
government staff
Building demand – emphasising learning, reports to Cabinet, working
with Parliament, publicising reports, training for DGs/DDGs in use of
evidence
Evaluation Repository – audit of evaluations since 2006, plus new - 103
evaluations on the website
2 provincial evaluation plans (WC and GP), working with 4 more
provinces, and 5 departments have departmental evaluation plans
First Annual Report on the National Evaluation System (for 2013/14)
now available
The Presidency: Department of Performance Monitoring and Evaluation
10
The Presidency: Department of Performance Monitoring and Evaluation
11
The Presidency: Department of Performance Monitoring and Evaluation
12
The Presidency: Department of Performance Monitoring and Evaluation
13
Early indications of policy impact




Role of improvement plans - produced for ECD, Land Restitution,
Grade R, CRDP, RECAP and Business Process Services.
First 6 month report for ECD, 2nd due (late).
Apart from the ECD evaluation, the next evaluation was only
commissioned in October 2012 – ie 18 months prior to this report. It
is therefore early days to see concrete impacts of the evaluations.
Early days, some examples of policy influence can already be seen:
 New ECD policy drafted responding to the ECD Diagnostic Review, including
the need to target children from conception;
 Renewed focus on quality of Grade R rather than just rollout as a result of
the Grade R evaluation – included in MTSF.
 Renewed focus on nutrition in children resulting from the evaluation of
nutrition interventions for children under 5, and a stunting target in the
Medium-Term Strategic Framework. The Improvement Plan will take this
further.
 Revised Business Process Outsourcing Scheme launched by Minister Davies
in UK
The Presidency: Department of Performance Monitoring and Evaluation
14
Grade R – short-term targets
The Presidency: Department of Performance Monitoring and Evaluation
15
Key challenges

Inadequate supply of skilled evaluators - range of skills
needed from proposal writing, evaluation
methodology, writing reports
 DPME working with universities and World Bank,
promote training of SPs

Insufficient PDI participants
 Changing panel criteria to promote access, but keep quality, have
30% PDI requirement in CFPs

Some evaluations are taking a long time to be brought
by departments to cluster (often 6+ months) and
thence Cabinet.
 DPME will bring if takes longer than 4 months.

Departments taking long (6 months+ delay) to finalise
improvement plans arising from evaluations, and to
report on progress.
 Include in MPAT standards and discuss with AG.
The Presidency: Department of Performance Monitoring and Evaluation
16
Challenges (2)

Weak programme planning
 Cabinet agreed need for support programme using
government/donor resources. Also develop MPAT standard to
drive change.

Insufficient funding for complex evaluations
 Programmes should reserve between 0.1% and 5% of
programme budgets for evaluation.

Some departments who are funding NEP evaluations
themselves are taking a very long time to procure
(notably DHS).
 These should be in departmental evaluation plans and
DPME procures for all evaluations selected for NEP. Need to
meet Minister DHS to discuss delays on 5 evaluations.

Some sectors/outcomes not proposing evaluations –
notably Health, Economic Infrastructure, Environment,
Public Service, Local Government.
 FOSAD suggested propose and 6 evaluations being proposed
to Cabinet on Crime, Health, Public Service
The Presidency: Department of Performance Monitoring and Evaluation
17
Taking the evaluation system forward
Undertaking audit of implementation
programmes in gov. – may well be 1000! If
each were to be evaluated every 5 years =
200/year – big challenge
 To do that extending national evaluation
system to provinces in 2014-15/2015-16 and
to departments from 2015-16. Is an issue of
capacity to support 46 national depts and 9
provinces.
 Focus on programme planning (again needs
resources)

The Presidency: Department of Performance Monitoring and Evaluation
18
Research
 Work starting on a diagnostic which will inform a
research strategy – balance between work we do on
actual research, vs support to influence the research
system so it provides research relevant to the
outcomes
 Some work being undertaken internally in DPME
 PSPPD supporting a research exercise in a number of
provinces and departments to see what is happening and
what role we should play
 Working to develop immediately a repository of key
research around each outcome
 Assignment on DG workload – starting
The Presidency: Department of Performance Monitoring and Evaluation
19
 Thank you
 Dr Ian Goldman
 Head: Evaluation and Research
 [email protected]
 Www.thepresidency-dpme.gov.za
The Presidency: Department of Performance Monitoring and Evaluation
20

similar documents