How we can use impact evaluation to assure effective use of resources for development

How we can use impact evaluation
to assure effective use of resources
for development
Maximo Torero,
m.torero@cgiar.org
Director
Markets, Trade and Institutions Division
(IFPRI)

IFAD-IFPRI Partnership, January 31st. 2012

Outline
1. Need for impact evaluation
2. Impact evaluation and monitoring
3. Guiding principles for our impact evaluation
approach
4. Impact evaluation concepts
5. Impact evaluation methods
6. Aggregating impact evaluation results
7. A methodology for external validity
8. Some examples
9. Final comments

Need for impact evaluation
 Helps identify and measure the results
 Helps identify the causal link between
intervention and results
 Provides a systematic and objective assessment
of program impacts
 Helps determine if interventions are relevant and
cost effective
 Promotes accountability, evidence-based
policymaking, and learning.

Need for impact evaluation
 Over past decade, increased demand from
governments, donor agencies and general
public, for evidence of Impact of development
policies.

 Political tool: Brings accountability regarding the use of
development money
 Fiscal tool / budgetary tool: Allocate resources across different
sectors or programs
 Management tools: Understand how to better reach the
objectives.

Monitoring and Impact Evaluation:
Monitoring
 A tool that provides regular information on:

 How a project is being implemented
 How a project is operating in the field
 How a project is progressing relative to targets
 What is the quality of service delivery (where applicable)

 Rationale for Monitoring:

 Provides basis for corrective action
 Holds implementers accountable for delivery of inputs
 Provides assessment of continued relevance
 Provides critical information for decision-making

Monitoring and Impact Evaluation:
Evaluation
 Impact Evaluation:

 Measures effectiveness and impact of programs or policies
on outcomes of interest

 Seeks to establish causality

 Not all programs need to be evaluated; not all outcomes
need to be measured in all evaluations

Indicators for Monitoring and
Evaluation
IMPACT Effect on living standards
- better welfare impacts (e.g literacy, health)
Evaluation

- increase in participation, happiness

OUTCOMES Access, usage and satisfaction of users
- e.g. school attendance, vaccination rates,
- food consumption, number of mobile phones

OUTPUTS Goods and services generated
- more local government services delivered
Monitoring

- e.g., textbooks, food delivered, roads built

INPUTS Financial and physical resources
- track resources used in the intervention
-e.g. budget support for local service delivery
Page 9

Guiding Principles of our IE Approach
1. Generate information to influence decisions

2. Specify which indicators and methods are most suitable for each
type of projects

3. Identify impact pathways

4. Evaluation activities must be built into the project design

4. Consider direct and indirect beneficiaries of projects

5. Evaluation at different levels of aggregation: Individual, thematic,
and overall program

6. Incorporate complementarities and substitution among project
impacts

Description of the Project
Stage 1: Stage 2: Stage 3:
Consultation Development Feedback

by Theme:
IFAD’s: •Technology • Governments,
• Objectives Identify: • Productivity • IFAD
• Activities • Indicators • Market Access • Implementers
• Information • Methods • Nutrition • CSO
needs
to
by level of Aggregation:
• Monitor performance • Individual projects
• Theme
• Evaluate Effectiveness
• Agricultural Development
• Asses Impact program

Target PEOPLE and vulnerable GROUPS:
• Poor and Women

Impact Evaluation: Impact Pathway

 The expected causal chain of events leading from project activities
to outputs, to changes in the target population, and to the
achievement of project objectives:

 From INPUTS OUTCOMES IMPACTS

 Focus on the impact pathway allows to:
 Understand how impacts are (or are not) achieved
 Allows generalizability of findings
 Provides key information for scaling up
 Identification of indicators for each step along the impact pathway

Illustrative Impact Pathway, Indicators, Methods
Example from: Science & Technology
IMPACT PATHWAY INDICATORS METHODS
Scholarships for plant breeders & Spending on scholarships
grants for agronomic research & research grants
PROCESS

 number and quality of varieties No. new varieties Internal program
released Approved & released monitoring

 availability and adoption of % male, female farmers
improved crop varieties Using improved varieties

Higher yields for farmers who Average yields among
adopted improved varieties adopting farmers
IMPACT

Intra-HH surveys:
Before/After,
Income, expenditure, Beneficiary/
 income,  poverty among farmer
Well-being indicators Control (Diff in
households
among target groups Diff)
(poor, women, etc.)

Applying the Methodology to specific
types of interventions
 Technology
 Example: Bio-fortification

 Productivity
 Example: Grants to crop breeding programs

 Market Access
 Example: Participation of small holders in the dairy value chain, “chilling
plant hubs”

 Nutrition interventions
 Example: Evaluation of specific interventions to improve nutrition of the
most vulnerable

Bio-fortification Project (Science and Technology)
Assumption: No price effect…

PROCESS

Bio-fortification Spending on bio-fortification
Internal program
R&D
monitoring
Adoption of bio-fortified varieties No. of farmers and land
adopting bio-fortified varieties.

Greater yields for farmers who Average yields among
adopted bio-fortified varieties adopting farmers

Production of bio-fortified varieties Total production of bio- HH surveys
fortified varieties •Beneficiary, control
• Farmers, consumers
Consumption of  consumption of No. of individuals and •DD estimator
bio-fortified animal products, average consumption (by •Randomization
varieties fruits, and vegetables type of individual))… •Panel: first round
effect vs. second
IMPACT

Reducing micronutrient round effects
Change in micronutrient status malnutrition
•Qualitative
Improvements in health, work Morbidity, mortality, information: two-way
performance, cognitive ability calling with the poor
enrollment ratio in primary

 income,  poverty among Income, expenditure,
farmer households Well-being indicators

Chilling Plant Hubs (Market Access)
PROCESS

Creation of farmer groups as dairy Internal program
Number of DFBA created
farmer business associations Monitoring
and number of farmers
(DFBA) participating (by gender)
Qualitative
Assessment:
Number of plants and milk organizational
Chilling plant construction
capacity capacity

Increase milk production of milk production of farmer
member farmers members

HH surveys
Reduction in loss through Volume of loss due to •Beneficiary, control
spoilage spoilage •DD estimator
•Non-experimental
IMPACT

design

Sales to formal markets and Value of sales to formal •Qualitative
traditional markets processors and to traditional information: two-way
markets calling with the poor

 income,  poverty among Income, expenditure,
farmer members Well-being indicators

M&E at Different Levels of Aggregation
Evaluation strategy Indicators Methods
What needs to be Cross theme Meta analysis
learned at the Program indicators: at the strategy
strategy level? level poverty level

Theme specific Meta analysis
indicators: within theme
What needs to be Theme  Market access
learned at the level Database at
 Productivity
theme level? project level
 Science and tech.
 Data Analysis
within themes

What needs to be Project Quantitative
learned at the Project level Indicators:
project level?  Process indicators
Qualitative
 Outcome
indicators analysis

Impact Evaluation:
Concepts

 Impact evaluation hinges on determining what would have
happened if the program had not existed.

 Good practice involves a comparison of outcome before and
after intervention with those with and without intervention

 Problem is identifying valid counterfactual

Impact Evaluation:
Methods

Quantitative Methods

cost

reliability
 Pre and post intervention, no control group
 Pre and post intervention, with control group, but
no randomization
 Pre and post intervention, with control group
and randomization

Qualitative Methods - complementary – help:
 Interpret of quantitative results
 Identify unexpected impacts, or effects on groups that are not captured
by quantitative surveys, etc.

Counterfactual

Ideally:
 Observe the outcome variable for those in the
program and
 For those same individuals had they NOT
participated in the program (the
counterfactual)
 So, constructing the counterfactual is the key
issue that any empirical method must
effectively handle.

Impact Evaluation: Finding a Counterfactual
Before the program After the program

A: “Treatment” Status
Beneficiaries: E: Status before
the program Real
B: “ Non Treatment” Status Counterfactual

C: “T reatment” Status
Non-Beneficiaries: F: Status before
the program
Estimated
D: “ Non Treatment” Status Counterfactual

Shaded boxes are Unshaded boxes are
Observable situations Unobservable

Concept: How is the outcome different than it would have been if the project
had not been implemented? = A – B (but cannot be observed)
So estimated impact is based on double difference: (A-E) – (D-F)

Supposed we observe an increase in outcome Y for
beneficiaries over time after an intervention

(observed)
Y1
Intervention

Y0

baseline(t0) follow-up(t1)

Page 24

To measure impact, we need to remove the counterfactual
from the observed outcome

(observed)
Y1 Impact=
Intervention Y1-Y1*
Y1* (counterfactual)

Comparison
Y0

baseline(t0) follow-up(t1)

Page 25

Treatment Effects: key obstacles

 Experimental vs. Non-Experimental Data
 Experimental data rules out self-selection into the
program (according to observables or
unobservables) as a source of bias in measuring
the treatment effect

 So, this contribution of experimental data brings
into high relief the two key obstacles that non-
experimental data methods must overcome in
order to avoid biased estimates of the average
treatment effect:

Treatment Effects: Key obstacles (cont)

 1. Self-selection into the program due to observables
characteristics
 2. Self-selection into the program due to unobservable
characteristics

 Accounting for #1 is often difficult (or impossible) to
accomplish.
 Even if #1 is accounted for in the method but # 2 is
not, then bias in the result will inevitably occur.

 Similarly if the control and treatment groups are randomly selected from
a population then there is no bias in the initial characteristics
 The impact of the procedure X can be attributed to the differences in
the variable Y between the control and treatment group.

Treatment Group
Random (receives procedure
Population
selection X)

Y Exp – Y Control

Control group
(not receives procedure X)

 Although normally experimental methods are not applied
 ¿Why we can not then apply a direct comparison between the control
and treatment group? Because differences in characteristics of
subjects, or what is called selection bias.

NO random Treatment Group
Population (receives procedure
selection
X)

Because of initial
differences between
both groups, the effects
of the treatment can not
be identified by directly
comparing the groups

Quintile I Quintile II Quintile III Quintile IV QuintileV Control group
(more poor) (more richer)

Selection bias: “Graphically”
Observed difference (G)
Impact on the treated (ATT) = true effect of the program on its recipients
Selection Bias (SB)

Observed SB = 0 SB > 0 SB < 0
G
G=ATT G>ATT G<ATT
No selection bias Selection on Selection on
“better-off” with “worse-off” with
respect to the respect to the
outcome outcome

Overcoming selection bias
 Ex-ante

 Experimental approach: the design of the program allows to introduce
randomness in its allocation

 Ex-post

 Natural experiment approaches: there are events that allow to simulate
“exogeneity in the choice of treatment”

 Control approaches: try to neutralize (reduce) as much as possible the selection
bias

Experimental approaches
 Randomly allocate “Treatment” into a population.
 Eliminates selection bias:

E  YiC | T  = E  YiC | C   SB  0, G  ATT
   
 Sometimes ethical critics
 If the exclusion of some beneficiaries is only due to the evaluation, while benefits
are well known
 In reality, resource constraints are the limiting factor. Then, random selection can
be considered a fair process (every potential beneficiary has same chance of
being selected)

 Must be designed before the start of the program

 Remains the best approach.

How to randomize?
 Randomize program as a whole.
 E.g. oversubscription: when there are limited supply and excess demand

 select recipients by lotteries.

 Randomize phasing-in
 Program cannot reach all intended beneficiaries the first year.

 select first year recipients randomly

 Randomize encouragement.
 Cannot randomize treatment for ethical or practical reasons.

 Randomly allocate encouragement (e.g. vouchers).
Only increases the probability that a treatment is received without
changing it from zero to one  specific analytical challenges
(partial (or imperfect) compliance).

Natural experiment approaches
 Use the fact that the program was allocated to some potential beneficiaries
and not to others, for reasons that have nothing to do with the outcome
itself.
 Find variable that is strongly linked to participation (fully or partially)
but not to outcome.

 Pipeline comparisons when administrative delays.
 Compare current participants to prospective participants who also qualify.

 Regression discontinuity when program selection based on clear
threshold on a given variable.
 Compare people just before threshold to people just above.

 Instrumental variables
 Use predicted participation as given by a variable linked to participation
but not to outcome

Limitations of These Methods
of Impact Analysis
 Impact evaluation focuses on program benefits, ignoring
costs. Measures one side of cost effectiveness.
 This limitation provides motivation for cost studies
(Caldés, Coady and Maluccio, 2004)
 Methods provide estimates of average impact in a ‘black
box’ form. Good for demonstrating impact, but limited for
broader policy analysis (Ravallion, 2005)

Page 36

Controls approaches
 Matching: compare people with similar ex-ante observable characteristics
 Control for the effects of observable characteristics that may affect hh
outcome.
 Assumption: All components of selection bias are observable and

measured (no omitted variables).

 Difference in difference: compare the evolution of the hh with treatment to
the evolution of the hh without treatment
 Neutralize time-invariant individual characteristics (observable and

unobservable).
 Neutralize effect due to other external events that may have affected

outcome since the program started.
 Assumption: absent the treatment, the outcomes in the two groups

would have followed parallel trends

 Mixed: difference in difference on matched households

Summary
Problem with “before / after” measure
Welfare
Difference could be driven by other events
measure

Problem with “with / without” measure

1’
1 Difference could be driven by selection

2
Double difference 1: “differences in evolution”
Impact = (1) – (2).
Controls for other events and self selection if the
latent heterogeneity is additive and time invariant.

Double difference 2: “differences in evolution”
With Without With Without Impact = (1’) – (2).
Before… After… Where initial differences are controlled for.
E.g. matching and difference in difference

 If randomization or natural experiment approach, then original differences should not exist. In
such cases, with/without measures can be sufficient

Different Levels of Aggregation:
A Common Evaluation Measure
∑ theme level +
complementarities –
Program level
substitution
(potential GE ERR,
PRRR
effects)

∑ project level +
complementarities - Theme level
ERR, PRRR
substitution

Careful evaluation
at this level is the Poverty Project Level:
Economic
Reduction
foundation of Rate of Rate of
- Program Logic Diagram
higher-level Return Return - Impact evaluation
evaluations ERR PRRR - Cash flow analysis

The concept of (stochastic) profit frontier to
assure external validity
 This approach is based on a
simple economic concept: the Milk
production
Production Possibility Frontier
(PPF).
Production
 All the possible production Possibility
Frontier
combinations are found within
the PPF.
 Outside of the boundary are
combinations which are not
C
achieveable under current
Corn
conditions production
 The efficient use of resources is
along the boundary.

Accessibility

Altitude

Water bodies

Roads

Land use

Advantages of Micro-Region Typology
Productive projects differentiated to Conditional Cash Transfers and
meet local needs and problems Nutritional Programs

The inclusion of
socioeconomic
characteristics and access in
What are the principal differences the analysis allows for the
between high and low efficiency identification of bottlenecks in
households in the area? areas of high potential but
low or medium efficiency

Productive and Efficiency
High potential and low Low potential and low potential based on market,
Typology average efficiency
socioeconomic, bio-physical
average efficiency and access characteristics.

Diagnostic
from High poverty areas High poverty areas
Poverty
map
44

Advantages of a micro-region typology:
classification

Micro-Regions Poverty Potential Efficiency
Critical, lacking agricultural potential High Low High-Medium-Low
Medium priority, no agricultural opportunities Medium Low High-Medium-Low
Low priority Low Low High-Medium-Low
High priority High Medium-High High-Medium-Low
Medium priority, with agricultural opportunities Medium Medium-High Medium-Low
Low priority, with agricultural opportunities Low Medium-High Medium-low
High performance Low Medium-High High

Estimation Methodology
Estimation inputs
INSUMOS PARA LA ESTIMACION ESTIMACION
Estimation Estimation output
OUTPUT DE LA ESTIMACIÓN

Potential
Potencial:   Precios de
Prices of products (P) and
productos (P) y insumos (W), Pesos asignados a los
Step 1:PASO 1: inputs (W), profits reported by Econometría Weights assigned to
ESTIMACION beneficios reportados por el Econometric Model insumos de acuerdo a
Estimation household (π). Modelo de inputs following
(NIVEL DEL
(Household hogar (π). of the teoría económica y
fronteras economic theory and

Level) HOGAR) stochastic
Efficiency Land, value of estocásticas de empirical evidence
evidencia empírica
Eficiencia :  tierra, valor de Profit frontier
activities, socioeconomic beneficios
los activos, características
characteristics (Z),
PASO 2:
biophysical conditions (G),
socioeconómicas (Z), Step 2:
PREDICCIÓN
Prediction
market access (A).
condiciones biofísicas (G), (NIVEL
(Regional
acceso a mercado (A). REGIONAL)
Level)

RESULTADO DE LA PREDICCION INSUMOS PARA LA PREDICCION
RESULTADO FINAL
Final result Prediction result Prediction inputs
Resultado de la estimación
Potencial productive a Estimation results (weights)
Productive potential at the (pesos)
nivel regional; eficiencia
regional level;
Potencial Boundary Product prices (P) and
Frontera:  Precios de productos
Region level de acuerdo a las
productive y Efficiency according to inputs (W)
productive (P) y insumos (W).
características
socioeconomic
potential and Pesos
eficiencia a
characteristics, biophysical
socioeconómicas, Efficiency: Land, value of
efficiency Eficiencia: tierra, valor de
nivel conditions, market access activities, socioeconomic
condiciones biofísicas,
within the area activos características
characteristics (Z), biophysical
regional
acceso a mercado dentro conditions (G), market access
socioeconomicas (Z),
(A).
del área condiciones biofísicas (G),
acceso a mercado (A).

Recap (1)… Targeting Criteria
Data
based on Efficiency

Estimated
Geo Layers cost of
Market
Access

PPF:
Input, Output, Profits
Agricultural
Profit Frontier

Variable X Variable Z
.4

1

Group 1

Available datasets:
Group 2

Efficiency in
.3

Land characteristics,
Agricultural
.9
Cumulative Density

biophysical conditions,
Density

Group 1
.2

Profits
Group 2

socioeconomic
.8

characteristics, assets,
.1

market access, etc.
.7
0

4 6 8 10 12 0 2 4 6 8 10
values X Values Z

Recap (2)…
MULTIPLE TARGETING
Efficiency Allocation

DIMENSIONS
Criteria

Typology
combines all
these criteria
Equity Allocation
Criterion

Recap (3)…

Recall the initial objective….

Micro-Regions
Critical, lacking agricultural potential
Low potential and low
average efficiency Medium priority, no agricultural opportunities
Low priority
High priority
High potential and low
Medium priority, with agricultural opportunities
average efficiency
Low priority, with agricultural opportunities
High performance

49

Recap (4): Grouping
diverse criteria into
seven
microregions…

Recap (5)… How does this translate into policies?

High potential and low
average efficiency

What are the principal differences
between high and low efficiency
households in the area?

Productive projects differentiated to
meet local needs and problems

Recap (6)… How does this translate into policies?

Low potential and low
average efficiency

Conditional Cash Transfers and
Nutritional Programs

Can be applied to other settings? Guatemala

Cost of Market Agricultural Profit
Access Frontier

Efficiency in Poverty Map
Agricultural Profits

Guatemala: Seven-Class Typology

With
agricultural
potential

Without
agricultural
potential

Some examples
 Extension services

 Market information

 Infrastructure in rural areas

 Property rights – land titling

Diff-in-Diff, FE (Dercon et al 2008)
 Impact of road quality improvements and increased access to
agricultural extension services on consumption and poverty in rural
Ethiopia.
 Dependent variables: household is poor, consumption growth
 Treatment: receiving at least one extension visit, and access to all-
weather roads (=1 if road to nearest town is all-weather road)
 Identification: IV model using GMM and controlling for household fixed
effects
 Instrument for consumption in time t-p: fertile land holdings, number
of adult equivalents and number of livestock units (all in logs) at time t-
p.
 Receiving at least one extension visit reduces headcount poverty by 10
percentage points and increases consumption growth by 7 percent.
Access to all-weather roads reduces poverty by 6.9 percentage points
and increases consumption growth by 16.3 percent.
 Ex post
 Supply driven

Institutional arrangement for a simple
price information system

Source: Hernanini (2007), World Bank

Flow of information and Institutional
agreements for virtual markets

Source: Hernanini (2007), World Bank

Pseudo randomized IV - Mobile phones: the impact of Cell
phones on grain Markets in Nigeria (Jenny C. Aker - 2008)

 Due partly to costly information, price dispersion across markets is
common in developed and developing countries
 Between 2001 and 2006, cell phone service was phased in
throughout Niger, providing an alternative and cheaper search
technology to grain traders and other market actors
 The author constructs a novel theoretical model of sequential search,
in which traders engage in optimal search for the maximum sales
price, net transport costs
 The model predicts that cell phones will increase traders’ reservation
sales prices and the number of markets over which they search,
leading to a reduction in price dispersion across markets.
 To test the predictions of the theoretical model, they use a unique
market and trader dataset from Niger that combines data on prices,
transport costs, rainfall and grain production with cell phone access
and trader behavior.
Page 63

Main results
 The results provide evidence that cell phones reduce grain price
dispersion across markets by a minimum of 6.4 percent and
reduce intra-annual price variation by 10 percent
 Cell phones have a greater impact on price dispersion for market
pairs that are farther away, and for those with lower road quality. This
effect becomes larger as a higher percentage of markets have cell
phone coverage.
 They provide empirical evidence in support of specific mechanisms
that partially explain the impact of cell phones on market
performance.
 The primary mechanism by which cell phones affect market-
level outcomes appears to be a reduction in search costs, as
grain traders operating in markets with cell phone coverage search
over a greater number of markets and sell in more markets.
 The results suggest that cell phones improved consumer and trader
welfare in Niger, perhaps averting an even worse outcome during the
2005 food crisis.
Page 64

Pseudo randomized IV: Internet: Internet kiosks in India to
provide wholesale price information (Aparajita Goyal, 2008)
 Beginning in October 2000, it set up 1700 internet kiosks
and 45 warehouses in Madhya Pradesh that provide
wholesale price information and an alternative marketing
channel to soybean farmers in the state
 Dependent variables: wholesale price of soybeans,
sales in traditional markets, soybean cultivation
 Treatment: presence of internet kiosks and price
warehouses
 Identification: variation in timing of the introduction of
kiosks and warehouses
 Equivalent to randomization at the village level
 Ex post
 Demand driven

Main results

 The estimates suggest an immediate and significant increase in
the monthly wholesale market price of soybeans by 1-5 percent
after the introduction of kiosks, lending support to the predictions
of the theoretical model

 While the presence of warehouses appears to have no effect on
price, warehouses are associated with a dramatic reduction in the
volume of sales in the traditional markets

 Moreover, there is a significant increase in the area under soy
cultivation. The estimates are robust to disaggregated measures of
treatment and comparisons with alternative crops grown in the same
season as soy

 The results suggest that information can enhance the functioning of
rural markets by making buyers more competitive.
Page 66

Pipeline comparisons when administrative delays (Torero
2008)
A B C D E
Table 1. Timeline

Secti Scheduled Start Scheduled End
on Date Date

The road to be improved was split A July 2008 June 2010
in the following segments: B Completed Completed
C July 2008 December 2009
D October 2008 April 2010
E October 2008 February 2010
Table 2. Treatment and Control Groups

Test Control Treatme
Number Group nt Group
Based on the geographic location and the
1 B A
timelines, the following treatment-control
groups are suggested:
2 B C

3 D C

4 E D

Pipeline comparisons when administrative delays

Pipeline comparisons when administrative delays.

Randomized- Barriers to connection in Ethiopia (Bernard and
Torero 2009)
 Connection fees range between USD 50
and USD 150 (drop down line and meter).
Need to find ways to facilitate connection
for the poorer.

 Can CFL (energy-saving light bulb)
positively influence energy use? How to
promote the use of energy-saving light
bulbs (consumes 4 times less, but costs 8
times more)?

 What is best: 2 years loan or 5 years loan
for connection fee?

 Pilot study on 20 towns to assess optimal subsidies.
 Experimental approach (randomize encouragement
through distribution of vouchers).

This image cannot currently be display ed.

Random selection…

Public distribution

Matching, IV on cross-sectional data - Land
property rights on productivity (- Markussen 2008)
 Dependent Variable: (log) value of output per
hectare
 Treatment: The plot is held with a paper
documenting ownership (titles, application
receipts)
 Identification: IV mode of plot acquisition
(dummies to indicate if the plot was given by the
State, inherited, bought, donated, occupied for
free) as instrument for the dummy “plot held with
paper”
 Ex-post
 Demand driven: households and landholders
apply for titles

Pseudo-Randomized - Land titling on rural
households (Torero and Field 2007)
 Dependent variables: household expenditure,
change in rent/market value of dwelling, risk of
expropriation, production, trade of land, collateral
and credit markets, land ownership and tenancy,
permanent ant transitory crops
 Treatment: to receive land title
 Identification: quasi random program
implementation, kernel matching
 Ex post
 Demand driven, but few requirements and
virtually free

The Database
The survey covered 3204
Peruvian rural households:
521 from rural coast, 1622
from rural highlands and
1061 form rural jungle.

The next map plots the
towns covered by the survey
and the valleys reached by
the PETT program. From
these 3204 households 1793
match with at least one
previous national survey.

 If control and treatment groups are randomly selected from a population
then there is no bias in the initial characteristics
 The impact over income can be attributed to the access to title

Treatment Group
Random (receives procedure
Population
selection X)

Y Exp – Y Control

Control group

 ¿Why we can not then apply a direct comparison between the control
and treatment group? Because differences in characteristics of
subjects, or what is called selection bias.

Pseudo Treatment Group
Population (receives procedure
random
selection X)

Because of initial
differences between
both groups, the effects
of the treatment can not
be identified by directly
comparing the groups

Quintile I Quintile II Quintile III Quintile IV QuintileV Control group
(more poor) (more richer)

 Identify comparable pairs (with similar initial characteristics) and that
differ only on the procedure
 We will use Propensity score matching.

Find the pair to assure
comparability Treatment

Population pseudo
random
selection

Impact of
the
procedure

Control

Final comments
 The impact evaluation must be a part of the program
design
 It is very important to identify how to incorporate it
now that the program already exists
 For new programs it is necessary to invest in the
design so that an impact evaluation is also part of
it
 It is essential to identify the impact pathways, i.e. the
expected causal chain of events leading from project
activities to outputs, to changes in the target population,
and to the achievement of project objectives

 Since the beginning of the program is necessary to
specify the expected “outcomes” and the control group

Final comments
 It will be ideal to have an autonomous and external
laboratory of impact evaluation

 Communications among all stakeholders is central

 Not all interventions need to be evaluated, it will be ideal
to do it before scaling up so there is assurance that the
intervention works
 Alignment of proper incentives – to contractors,
evaluators, to implementers and to USAID country
offices
 Finally, policy requires a causal model; “without it, we
cannot understand the welfare consequences of a
policy” (Deaton 2009)

Recommended readings
Caldés, Natalia, David Coady and John Maluccio. 2004. The Cost of Poverty
Alleviation Transfer Programs: A Comparative Analysis of Three Programs
in Latin America. IFPRI FCND Discussion Paper No. 174., Washington, DC.
Duflo, Esther; Rachel Glennerster and Michael Kremer (2007):“Using
Randomization in Development Economics Research: a Toolkit”CEPR
discussion paper no. 6059
Feder et al. 2004. Review of Agricultural Economics.
Godtland et al. 2004. Economic Development and Cultural Change.
Heckman, J.J., H. Ichimura, and P.E. Todd. 1997. “Matching as an
Econometric Evaluation Estimator: Evidence from Evaluating a Job Training
Program.” Review of Economic Studies 64:605-654.
Hirano and Imbens. 2004. The Propensity Score with Continuous Treatments.
In Gelman & Meng, eds.
Miguel and Kremer. 2004. Econometrica.
Ravallion, Martin. 2005. Evaluating Anti-poverty Programs. World Bank
Working Paper Series 3625, Washington, DC. Martin Ravallion (2003): “The
Mystery of the Vanishing Benefits: An introduction to Impact Evaluation”
The World Bank Economic Review, volume 15, no 1, pp115-140
Page 85

How we can use impact evaluation to assure effective use of resources for development

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (18)

Similar to How we can use impact evaluation to assure effective use of resources for development

Similar to How we can use impact evaluation to assure effective use of resources for development (20)

More from IFAD International Fund for Agricultural Development

More from IFAD International Fund for Agricultural Development (20)

Recently uploaded

Recently uploaded (20)

How we can use impact evaluation to assure effective use of resources for development