LAB 8 - Introduction To Impact Evaluation

PRAKTIKUM LAB 8
INTRODUCTION TO
IMPACT EVALUATION
Teaching Assistant
Microeconometrics 2022
bit.ly/mitrik2023-lab8
MAIN TOPIC
Impact Causal Counter

Evaluation Inference Factual
Impact Evaluation
WHAT IS
IMPACT EVALUATION?
Impact evaluations are a particular type of evaluation that
seeks to answer cause-and-effect questions. Unlike general
evaluations, which can answer many types of questions, impact
evaluations are structured around one particular type of
question: What is the impact (or causal effect) of a program on
an outcome of interest? (World Bank, 2011)
“How effective are these programs in changing those economic

or social outcomes?”
PURPOSE OF
IMPACT EVALUATION
01. Measure the impacts of programs
and policies on outcomes of interest
04. the
Develop a stock of knowledge on
effectiveness of various
projects, programs and policies
02. improving
Provide useful
existing
information
programs
for
and
(how works and how much to
spend)
policies
05. effective programs and policies
Generate political support for
03. evaluations,
Based on results
or on
from
estimates
other
of
behavioral models, predict
impacts of proposed policies and
the
06. government
Generate political support for the
programs
Causal Inference
The basic impact evaluation question
essentially constitutes a causal inference
WHAT IS problem.
We can say that,
CAUSAL
“Assesing the impact of program
on a series of outcomes”
is equivalent with,
INFERENCE? “Assesing the causal effect of the

program on those outcomes.”
Most policy questions involve cause-and-effect
relationship:
• Does teacher training improve students test
score?
• Do conditional cash transfer programs cause
better health outcomes in children?
• Do vocational training programs increase
trainees income?
The answer to the basic impact evaluation question:
α = (Y | P= 1) – (Y | P= 0)
α : Causal impact
Y : Outcome
(P= 1) : With program
(P= 0) : Without program/represent counterfactual
THE PROBLEM OF
IMPACT EVALUATION
The main problem of Impact Evaluation is:
"What their outcomes would have been if they had not
participated."
This is often referred to as the:

Counterfactual Outcome or simply known as, Counterfactual
Known Unknown
Performance if she has worn Performance without glasses
glasses a long time
For a student who has been Even the performance of her twin
wearing eyeglasses for a long sister is not a very good
time, we don’t know what her counterfactual, because even
academic performance in the identical twins might experience
state in which she never wears different learning environments
eyeglasses
Counterfactual talk about:
“What would happened without the program?"
WHAT IS The problem is that we never observe both

(Y | P = 1) dan (Y | P = 0) at the same time. In
particular, we can observe (Y | P = 1) for
COUNTER program participants but we cannot observe

(Y | P = 0) for participants.
FACTUAL? Since (Y | P = 0) cannot be directly observed

for program participant, hence we get the
information by estimating the counterfactual.
To estimate oounterfactual, use comparison HOW TO
group (sometimes called “control groups”)
since there’s no perfect clone exists for a
single individual.
ESTIMATE
Statistical tools exist that can be used to
COUNTER
FACTUAL?
generate two groups.
CONTRUCTING THE
COUNTERFACTUAL
Usually done by selecting a group of
individuals that did not participate in the
program
This group is usually referred to as the control

group or comparison group
How this group is selected is a key decision in

the design of any impact evaluation
KEY CONCEPT
A valid comparison group will When the comparison group for
have the same characteristics an evaluation is invalid, then the
as the group of participants in estimate of the impact of the
the program (treatment group), program will also be invalid:
except for the fact that the units
in the comparison group do not it will not estimate the true
benefit from the program. impact of the program. In
statistical terms, it will be
“biased”
HOW TO CHOOSE
"COMPARISON GROUP"
Before we determine the evaluations design, we choose the method first to
identify a comparison/control group
Impact Evaluation Method
IMPACT EVALUATION METHOD
Experimental Method Non or Quasi Experimental
Randomized Evaluations
Use random assignment of the program to • Pre-post
create a comparison group which mimics the • Simple difference
counterfactual. • Difference in difference
• Statistic maching
Also known as: • Intrupted time series
• Random Assignment Studies • Multivariate regression
• Randomised Field Trials • Instrumental variable
• Social Experiments • Regression discontinuity
• Randomised Controlled Trials (RCTs)
• Randomised Controlled Experiments
RANDOM ASSIGNMENT TO THE TREATMENT
Comparison
Population
Evaluation
sample
Treatment
External Validity Internal Validity

The experimental sample The control group provides
is representative of the a valid counterfactual for
target population the treatment group
THANK YOU!
Sources:
• Gertler, Paul J et al. 2011 The World Bank Publications Impact
Evaluation in Practice. The International Bank for Reconstruction and
Development.
• Djebbari, Habiba, and Maria Adelaida Lopera. 2014. “Impact
Evaluation Using Stata.” Impact Evaluation in Practice: 1-30
• Bahan Ajar Impact Evalution oleh Tim Dosen untuk Kelas
Mikroekonometrika
• Bahan Ajar Impact Evaluation oleh Pak Nur Cahyadi (TNP2K)

LAB 8 - Introduction To Impact Evaluation

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

LAB 8 - Introduction To Impact Evaluation

Uploaded by

Copyright:

Available Formats

PRAKTIKUM LAB 8

Impact Causal Counter

“How effective are these programs in changing those economic

We can say that,

INFERENCE? “Assesing the causal effect of the

• Does teacher training improve students test

• Do conditional cash transfer programs cause

better health outcomes in children?

• Do vocational training programs increase

This is often referred to as the:

WHAT IS The problem is that we never observe both

COUNTER program participants but we cannot observe

FACTUAL? Since (Y | P = 0) cannot be directly observed

This group is usually referred to as the control

How this group is selected is a key decision in

External Validity Internal Validity

You might also like