An evidence review is a critical review of secondary sources related to your program. It helps you understand what information is available about the problems you’re trying to address and your program design: What interventions have already been tried? In what context? Were they successful? Why or why not?
An evidence review helps you build on top of programs with a proven track record of effectiveness, leverage existing resources, and avoid strategies that have already been proven ineffective – saving you time and resources in the long run. It can also strengthen your theory of change by helping you understand what assumptions are reasonable, given the existing evidence base.
Say you want to start a malaria prevention program. You are wondering whether to focus on spraying pesticide to kill mosquitoes or distributing insecticide-treated bed nets. You conduct an evidence review, and learn that multiple studies have confirmed that sleeping under bed nets leads to decreased rates of malaria. You also note that the contexts in which these studies took place are similar to the context where you plan to implement the program. Based on this evidence, you decide that your program will distribute bed nets.
Which questions can an evidence review answer?
An evidence review can save you significant time and effort by helping you build on previous successes and avoid past failures. You can model your program after what has worked in similar contexts, or avoid programs that have not been impactful.
Even if your program includes proven interventions, you’ll still need to ensure they are implemented well and that they are appropriate for your context:
- If you cannot deliver the program well, it would not matter if the program has a track record of effectiveness. For example, if, based on an evidence review, you make low-interest loans available to women, but program staff forge records and pocket the money themselves, you will not see the expected impact
- A program that was successful elsewhere may fail in your context if key conditions are different. For example, a program that built schools in rural Afghanistan was found to be highly effective in raising student test scores. But the same program delivered in Peru or India, where most rural communities already have schools, would be unlikely to effect change.
An evidence review can tell you which components of a program drive its impact. You can focus your efforts to ensure that these parts are working well in your program. For example, say you are providing treatment for tuberculosis. Your evidence review shows that while antibiotics are a proven treatment, patients not adhering to the medication regimen is a big problem. Based on this, you may decide to monitor patients’ treatment adherence.
 Burde, D., & Linden, L. L. (2013).Bringing education to Afghan girls: A randomized controlled trial ofvillage-based schools. American Economic Journal: Applied Economics, 5(3),27-40.
Next steps - Conducting an evidence review
There are two steps for conducting an evidence review:
- Searching for relevant sources
- Reading through, understanding, and summarizing the research.
Should you do the evidence review yourself or hire an expert?
Most organizations can do an evidence review on their own. You may want to hire an external party to do an evidence review for the following reasons:
- Your team does not have the time to search for and read through the relevant sources
- Your team does not have prior experience consuming research
- You would like a more objective summary from someone who is not heavily involved in program implementation
How to conduct your evidence review
Step-by-step instructions and template assessment
There are three major steps for conducting an evidence review:
You first need to make a list of evidence that looks like it might be relevant. Later, you’ll go through this longlist in more depth to identify what is actually useful for you. This “longlist of evidence” is called an annotated bibliography.
In this step, we’re going to build an annotated bibliography. You’ll end up with a list of all sources you considered and skimmed, along with your quick takeaways for each source.
We suggest making the annotated bibliography in a spreadsheet using Google Sheets or Excel – have a look at this template to see which fields you should include. You may also use bibliography generators like Zotero, Mendeley for this purpose.
Now, onto finding sources for your bibliography. You can follow either (or both) of these two ways to identify evidence to add to the annotated bibliography:
We suggest reading through both methods before deciding which one you plan to implement.
Now that you have your annotated bibliography, it’s time to start going through the sources listed. You’ll ultimately synthesize your findings across sources and analyze their implications for your program. We suggest starting a Word document for recording your takeaways and beginning the synthesis work.
For each source in your bibliography, add useful information to your Word document after thinking critically about:
1. The quality of evidence
GRADE provides a standard framework for assessing whether the quality of the evidence is high, moderate, low, or very low.
Other considerations can include:
- Topic: What is the source about? How similar is the program(s) described in the source? Is it implemented at the same level – individual, household, village, school, etc.?
- Geography: Region or country, developing or high-income countries
- Type of study: Randomized control trial, representative survey, observational study, meta-analysis
- Results: What did the study find?
- Limitations: Are the any issues/drawbacks from the study? For example, estimates may be biased due to non-experimental methods, or non-random samples. Estimates could also be imprecise due to small sample size
- Source: Peer-reviewed journal, policy brief, research report
2. Whether the evidence is likely to apply to your context
The generalizability framework proposed by J-PAL is an excellent tool for understanding whether evidence from another context would apply well to your program.
Some general questions to consider are – did these studies measure similar outcomes? Were they conducted in a similar context (geography, target population, time period)? Were the programs similar to your program? What other contextual aspects make you confident in the validity of the study to the client context?
3. How the evidence helps you understand your program
Usually, useful evidence will do one or more of the following:
- Provide the scope of the problem, through statistics about its prevalence and case studies that indicate why it needs to be addressed
For example: Learning outcomes in government schools in Bihar are poor and stagnant, resulting in students being inadequately prepared for the job market.
- Enumerate the major constraints to solving the problem
For example: Supply-side constraints (physical inputs, teachers, administration, rigid curriculum and automatic promotion to next grade) and demand-side constraints (beliefs about the returns to education, opportunity costs of keeping children in schools)hinder progress on learning outcomes.
- Enumerate major attempts to solve this problem and how successful they have been
For example: Attempts to improve learning outcomes have either been school-focused (teacher motivation, facility improvement) or community-focused (information campaigns).While teacher motivation campaigns have been successful at improving learning outcomes, the frequency of the campaign and channels used to reach teachers greatly affect the success rate.
- Illuminate some of the links and assumptions in your theory of change
For example: Student enrollment for this district is 98%, but attendance is only 43%, so our program targets attendance
Now that you have reviewed the evidence, you should need to write out your findings.
Your evidence review should lay out your proposed program, findings from previous studies, and gaps:
Your proposed program
- Explain the theory of change for this approach to solving the policy problem. What are the major links and assumptions?
- Within the context of the theory of change, illuminate some of the links and assumptions. If possible, use data and qualitative evidence to strengthen the theory of change (e.g. 98% of children are enrolled in school, but average attendance is only 43%, so the program targets attendance).
Findings from previous studies
Describe previous studies that have evaluated similar programs. How compelling is the evidence from these studies? Evaluate these studies keeping the following considerations in mind:
- Relevance to your program: Did these studies measure similar outcomes and were they conducted in a similar context (geography, target population, time period)? How similar is the program? What other contextual aspects make you confident in the validity of the study to your program’s context?
- Rigor: Are the estimates from other studies statistically significant? Precise? How robust was the methodology to potential bias? Was high-quality data analyzed?
Describe the major gaps that remain in the theory of change, particularly any links or assumptions that lack evidence from a similar context to your program.