Beyond Average Impacts: Understanding Why Policies Work (Or Not)

This free public lecture is part of the What Works Global Summit.

Over the last few years much progress has been made in evaluating policy initiatives, in particular in developing countries. Thousands of Randomized Control Trials (RCT) are run and even when evaluators do not use what is now considered the gold standard of evaluation, they think carefully about attribution and causality. The next challenge is to go beyond the simple measurement of impacts to get to an understanding of the mechanisms that generate them. This is key to extrapolating the results to different contexts and different policies. Many interventions work because they fix specific frictions or market failures. It is imperative to identify which.

The identification of the mechanisms is done through the use of behavioural models. RCT are particularly useful in this respect because they allow the identification of a more realistic model through less stringent hypotheses. Rich and innovative measurement is the other dimension in which much progress can be made. These arguments will be developed with a few examples where RCT and structural models are combined to deliver a richer and more useful narrative about the impact of specific interventions.

Economists researching toilets: why and how?  This presentation will use EDePo’s sanitation work as an example of how a research agenda developed (and in fact is  developing); why we focus on specific questions and how we use rigorous impact evaluations as a basis to go beyond asking about average impacts to address additional policy relevant questions: What are the barriers to increasing sanitation coverage? How can we overcome these constraints? How effective are government subsidies in removing financial constraints? Is there any point having a toilet if your neighbours don’t? We will also discuss the role that implementing partners played and how we continue to engage with them beyond the basic evaluation.

That’s interesting, but how can we measure it?  Measuring what we are interested in, whether it be income, mental health or the cognitive development of a two-year old, is a fundamental challenge of policy analysis in developing countries. Inaccurate or unreliable measurements prevent us from being able to detect true effects of policies.  Questions and tools that behave very differently in different countries or communities prevent us from being able to compare the effects of policies across settings. We discuss these challenges and ideas for obtaining high quality measurement in large scale research.

It’s not just how or what but who:  Most evaluations look at what impacts we achieve and how, rather than asking whom we reach with our programmes.  Who do we reach, who is left behind, and why? Are we reaching the people we are intending to reach? Do we reach groups beyond those explicitly targeted, such as siblings, neighbours, relatives or friends? In this session we will present a selection of recent and on-going EDePo projects that aim to look at such questions related to targeting and spillover effects, in the context of community-based- development, sanitation and early childhood development in particular.