IDi-Logo-Master-RGB-White-1-1.png

Impact Measurement Guide Beta

About the theory of change
Example
Why you should develop a theory of change
How to draft a theory of change
Case study ->

About the theory of change

Every development program rests on a theory of how and why the program will work and lead to social impact. A theory of change formalizes this, representing how a program will use resources to conduct activities that can lead to changes in behavior and improvements in people’s lives. 

Whether you are designing programs, tracking program implementation, or measuring impact, a theory of change lays out the steps necessary to achieve your goals. It is also the blueprint based on which you will decide which evidence-generation tools to use. We thus strongly encourage all organizations to map out a theory of change for each of their programs to ensure teams have a shared vision of success. A theory of change is a living document that should be maintained – we also encourage teams to regularly revisit their theory of change, to make sure it reflects their current understanding of the program and its context.

Example

A very simple theory of change for a business selling solar lighting systems might be:

My program will help increase incomes by selling solar home systems that allow households to have light and electricity in the evening. [1]

Represented in a diagram, that same theory of change might look like this:

Even this simple narrative helps us frame important questions about whether the program is likely to be successful:

  • Do we know if households that lack electrification are interested in buying solar home systems?
  • Are we confident that the solar home system provides more light than kerosene lamps? 
  • Do households actually use the extra hours of light for studying and income-generating activities?
  • Is studying supposed to increase income, or are educational outcomes a separate objective not yet explicitly stated?


Note that while even a simple theory of change helps frame important questions, the theory of change you develop should reflect your program and will often be more complex.

[1] Loosely based on the theories of change for home solar providers d.light and M-KOPA

Why you should develop a theory of change?

A theory of change encourages critical, creative, and empathetic thinking on how the world works and how programs may work, or fail to work, in that world.

When all program activities and expectations about your impact are written out in a theory of change, it is easier for program staff to align on how the program should run. For instance, in our solar home program, staff may face tough decisions about whom to prioritize: households that have an at-home business or households with school-aged children? We have found that it is quite common for an organisation’s members to not be aligned on the priority objectives of their program. And when left implicit, this causes confusion. Developing a theory of change helps your team articulate these questions and make decisions in keeping with your organization’s values and goals.

A theory of change allows you to clearly describe the pathways that will lead your program to success – which, in turn, leads to a better-designed program. For example, a program to reduce malaria by distributing bed nets rests on multiple assumptions, including that recipients will use bed nets correctly after receiving them. Documenting this through a theory of change may lead you to realize that you need to investigate whether and how the bed nets are used after distribution, and possibly do more to explain their importance to recipients.

A theory of change will help you identify the steps that are essential for your program to be successful and should, therefore, be tracked with data. Using the previous example of the solar homes program, a theory of change may reveal that you are assuming that recipients do not have existing cheap sources of light, or need extra hours of light, and that is why they demand solar lights. Since this assumption is critical to the success of your program, you decide to measure recipients’ demand for light and check how it relates to purchases. Thus, by serving as a concrete representation of your program’s steps and assumptions, a theory of change can help you figure out where you need more evidence.

 

All the methods in the Impact Measurement Guide rely on a theory of change as a starting point, from deciding which assumptions your program can safely make based on existing research (evidence review), to assessing the strength of other assumptions (process evaluation), to deciding which critical activities you need regular data on (monitoring). This is also why a theory of change must be revisited regularly, to make sure it is in sync with the reality on the ground.

Documenting your program activities and the steps by which these activities lead to impact can help you better describe your program to donors and supporters.

How to draft a theory of change

To develop a draft Theory of Change follow the steps below:
  1. What is the goal that the program wants to achieve? 
    This is the long-term objective or “ultimate outcome” of the program. Your program might strive to achieve some improvement in livelihood related to education, health, and economic outcomes of individuals.  Example: A program that reminds parents to get their children vaccinated against malaria at upcoming vaccination camps may list “Improved children’s health outcomes” as its ultimate outcome.
  2. What does the program do?
    These are the activities directly conducted as part of the program. You should be as comprehensive as possible when listing the activities. Even preparatory activities (e.g., getting the list of households without vaccinated children) should be included. For clarity, it is also helpful to identify the personnel/stakeholder doing the activity.  Example: The vaccination reminder program may identify “Program Team [Who] sends the reminder text about the vaccination camp [does what] to parents [with whom/with which frequency]” as an activity.
  3. What are the products or services directly stemming from the program? 
    These are the outputs of the activities that you listed earlier. For each activity, ensure you understand what output they relate to. Oftentimes, outputs may feel redundant to program activities. However, the link between conducting the activities and the outputs being achieved can break for a variety of reasons. It is therefore useful to separate the two. Sometimes, outputs can be passive restatements of activities. Example: The vaccination reminder program may identify “Text reminder about the vaccination camp against malaria is sent to parents” as an output.
  4. What are the real changes in people’s lives that result from the achievement of the outputs? What will lead to the achievement of the goal of the program?
    Identify the outcomes that are directly linked to the outputs that you listed. These may be receipt of the service by the target population, such as “parents receive text reminders”. They are often changes in people’s knowledge, attitudes, behavior, and/or living conditions.  Example: The vaccination reminder program may identify “Parents take children to the vaccination camp” as a direct outcome of the output from the earlier example. To link that outcome to the overall goal, the program may identify “Children are vaccinated against malaria” as a final outcome.

These boxes are called the nodes of the ToC. Refer to the section below on structuring these nodes in a systematic manner. It may take several iterations until you and your team find a structure that works. Generally, we suggest to work with the following norms: 

  1. Arrange the TOC left to right (horizontally) to capture the program from inputs to impact. 
  2. Try to capture different stakeholders and/or work streams in different rows if possible. For example, a program might engage different stakeholders through policy-advocacy as well as direct implementation in communities, with frequent interaction between the two (e.g. through sharing of case studies). Arrange these two work-streams below one another. In that way you will be able to show how the work-streams interact. 
  3. Indicate the time component at key junctures. Are long term outcomes expected to take 5 minutes, months, or years to manifest at the goal levels?

These arrows represent causal links, i.e., the node the arrow starts from directly causes the node the arrow ends in (activities ➞ outputs ➞ outcomes ➞ goal). 

a. Be sufficiently detailed while also being concise.

Review the linkages in the ToC and verify that there are no missing nodes in its causal chains. If there seem to be too many nodes, review the identified activities and outputs. Are all of them essential to the ToC? Are there activities that are similar enough and can be grouped together as part of a larger workstream? How about the outputs? 

b. Be specific. 

Descriptive and precise nodes are more valuable than general nodes. For example, “Communications Team translates the reminder text to Hausa” is a better activity statement than “Reminder texts are translated”.

c. Ensure that statements follow proper syntax. 

Activities for example should always be formulated in active tense. Outputs are generally phrased in passive tense. Additional guidelines on syntax  that may further improve the clarity of your ToC can be found in the guidance document from Global Affairs Canada. 

d. Label the nodes with alpha-numbers so that they can be easily identified later on.
You could just use numbers to label nodes. In that case, minor changes to the ToC may require you to renumber all nodes, though. For this reason, we recommend that you use an alpha-numeric numbering scheme (A.1,A.1.1,A.2, etc.) to label nodes and linkages. 

e. Visualize feedback loops by linking downstream nodes back to those further upstream.
Feedback loops are generally important in programs that  take systematic approaches to broad problems. Feedback loops allow you to visualize the complexity and interactions of program components. For example, creating an alumni community could link back to student recruitment for a private skills training program. 

CategoryDefinition and Notes Examples 
Input 
  • The financial, human, material and information resources needed to deliver the program successfully and achieve the desired outputs and outcomes
Funding, people, material 
Activity 
  • Actions taken or work performed through which inputs are mobilized to produce outputs.

Active verbs such as: procure, hire, build, monitor 

  • Design gender sensitive training material
Output
  • Direct products or services stemming from the activities of a program
  • Water and sanitation facilities built/refurbished in rural areas of country X 
  • Community volunteers (f/m) trained to disseminate key messages on essential nutrition and hygiene actions in village Y, X, and Z of country X
Immediate Outcome
  • A change that is expected to occur once one or more outputs have been delivered by the program 
  • In terms of time frame and level, these are short-term outcomes, and are usually changes in capacity, such as an increase in knowledge, awareness, skills or abilities, or access to… among the target population of the program.
  • Immediate outcomes represent the first level of change that the target population experiences once the program starts delivering the outputs of a project.
  • Improved knowledge of sustainable agricultural-production practices among women-smallholder farmers in village X, of country Y
  • Increased ability of health workers to address the nutrition challenges of women and children, especially girls in county Z
Intermediate Outcome
  • A change that is expected to logically occur once one or more immediate outcomes have been achieved. 
  • In terms of time frame and level, these are medium-term outcomes that are usually achieved by the end of a program, and are usually changes in behaviour, practice or performance among the program’s target population.
  • Improved use of essential maternal health services, including those related to sexual and reproductive health, by women in village Y of country X
  • Enhanced equitable access to safe, quality education for girls and boys in crisis-affected province Y of country X
  • Increased use of business development and financial services by micro enterprises, particularly those led by women, in province Y of country X
Ultimate or Final Outcome 
  • The highest-level change to which a program contributes through the achievement of one or more intermediate outcomes. 
  • The ultimate outcome usually represents the raison d’être of an organization, policy, program, or project, and it takes the form of a sustainable change of state among beneficiaries.
  • An ultimate outcome usually occurs after the end of the project, but should, when feasible, still be measured during the life of the project as changes may occur earlier.
  • Enhanced economic prosperity for the poor, particularly women and youth, in country X
  • Increased food security of food insecure populations in region Y of country X
  • Improved equitable health of girls and boys under age five in rural areas of region X
  • Improved equitable learning outcomes of all girls and boys in crisis-affected province Y of country X
  • Increased freedom of marginalized women, men, girls and boys in country X
  • Enhanced well-being of women in village Y of country Z

Note: This table leverages the definition from Global Affairs Canada (GAC) as defined in the results-based management guidelines found here.

Next steps

A well-designed theory of change is a crucial first step in developing a program. Any organization can develop a theory of change – critical thinking and a deep knowledge of the program and context are all that is required. 

Guide

Not sure where to go from here? 
Use our guide to frame a question and match it to the right method.

Theory of change case study

How a theory of change helped set clear monitoring and evaluation priorities for a handwashing program in the Philippines

IDi-Logo-Master-RGB-White-1-1.png

Impact Measurement Guide

Twitter Linkedin Instagram Facebook

Feedback

The Impact Measurement Guide is still in beta release, we welcome your feedback on which features you like as well as what we should change or improve in future versions.

Feedback Form ->

Information

  • About
  • Terms & Conditions
  • Privacy Policy

Contact us

  • guide@idinsight.org
  • SALE ENDS IN:
Days :
Hours :
Minutes :
Seconds

— NEXT ONLINE COURSE STARTS ON MAY 26TH —

UI/UX Fundamentals
25% OFF

Enroll Now

Types of impact evaluations

Impact evaluations compare the people who receive your program to a similar group of people who did not receive your program. Based on how this similar group is chosen, impact evaluations can be randomized controlled trials or quasi-experimental.

A randomized controlled trial (RCT) is considered the gold standard of impact evaluation. In an RCT, the program is randomly assigned among a target population. Those who receive the program are called the treatment group; those who do not receive the program are called the control group. We consider the outcomes from the control group to represent what would have happened to the treatment group in the absence of the program. By comparing outcomes among those who receive the program (the treatment group) to those who don’t (the control group), we can rigorously estimate the impact of the program. The random assignment of the program to the treatment and control group provides the rigor, as it ensures that the selection of people is not based on biased criteria that could affect the results.

When a randomized design is not feasible, there are other, “quasi-experimental,” ways of constructing a valid comparison group. 

  • In a matched design we would match individuals who receive the program to individuals who don’t receive the program based on some observable characteristics (such as age, gender, number of years of schooling, etc.), and compare outcomes across these groups. 
  • Another common technique is regression discontinuity design, in which you create a cutoff based on which individuals are eligible to receive the program, and then compare outcomes from groups just below and just above the cutoff to estimate impact. 

Matched designs and regression discontinuity designs are just two of many quasi-experimental techniques. J-PAL provides an overview of common methods of conducting an impact evaluation. All such methods seek to identify what would have happened to your target population if they had never received the program, and their success relies on the strength of the assumptions they make about whether the comparison group is a credible stand-in for your program’s target population. 

Recommendation: Theory of Change

A theory of change is a narrative about how and why a program will lead to social impact. Every development program rests on a theory of change – it’s a crucial first step that helps you remain focused on impact and plan your program better.

You can use diagrams.net to create your Theory of Change.

Learn more about the theory of change ->
Go to the next state of the decision guide

Recommendation: Needs assessment

A needs assessment describes the context in which your program will operate (or is already operating). It can help you understand the scope and urgency of the problems you identified in the theory of change. It can also help you identify the specific communities that can benefit from your program and how you can reach them.

Build your theory of change ->

Once you’re satisfied that your program can be implemented as expected:

Go to the next state of the decision guide

Your program's theory of change

You have a written plan for how your program will improve lives – great! Make sure to refer to it as you explore the different sections in the Impact Measurement Guide, as it is the foundation for any other method you’ll use.

Learn more about the theory of change ->
Go to the next state of the decision guide

Recommendation: Process Evaluation

A process evaluation can tell you whether your program is being implemented as expected, and if assumptions in your theory of change hold. It is an in-depth, one-time exercise that can help identify gaps in your program.

Learn about process evaluations ->

Once you are satisfied that your program can be implemented as expected:

Go to the next state of the decision guide

Recommendation: Evidence Review

An evidence review summarizes findings from research related to your program. It can help you make informed decisions about what’s likely to work in your context, and can provide ideas for program features.

Learn about evidence reviews ->

Once you are satisfied with your evidence review:

Go to the next state of the decision guide

Recommendation: Monitoring

A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.

Learn more about monitoring ->

Once you are satisfied with your monitoring system:

Go to the next state of the decision guide

Recommendation: Monitoring

A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.

Learn more about monitoring ->

Once you are satisfied with your monitoring system:

Go to the next state of the decision guide

How to compile evidence, method 2

You’re on this page because you want to search for evidence relevant to your program.

Here are some academic sources where you can search for relevant research:

  • Google Scholar is a search engine for academic papers – enter the keywords relevant to your program, and you’ll find useful papers in the top links
  • Once you identify some useful papers, you can consult their literature review and bibliography sections to find other papers that might be relevant
  • Speaking to a sector expert can guide you to useful literature

 However, don’t include only academic studies in your review! You should also consult:

  • Policy reports
  • Websites of organizations involved in this issue, such as think tanks, NGOs, or the World Bank
  • Public datasets
  • Your program archives – data and reports from earlier iterations of the program can be very valuable!

 The free Zotero plug-in provides an easy way to save, organize, and format citations collected during internet research. Note that Zotero can help you start your annotated bibliography, but it is not a substitute since it does not include any summary or interpretation of each study’s findings.

How to compile evidence, method 1

Start with the 3ie Development Evidence Portal, which has compiled over 3,700 evaluations and over 700 systematic evidence reviews. Steps 2-7 of this example are specific to locating evidence on the 3ie website, but you can also consider looking for a review by J-PAL or Campbell Collaborations or the Cochrane.

For example, suppose your goal is to increase immunization rates in India. Type “immunization vaccination” or other related terms into the search box, and click the magnification lens to search.

The search results include individual studies, which are usually about a single program in a single location, as well as “systematic reviews”, which is what we are looking for because they are more comprehensive. To show only the systematic reviews, on the left of the screen under Filter Results, click on PRODUCTS and check the Systematic Reviews box. We’re now left with 17 evidence reviews related to immunization.

Now you might want to further narrow your search by region or country. In our example, suppose we want to see only those evidence reviews that contain at least one study from India. Click on COUNTRY and scroll down to click on India.

There are still 9 evidence reviews! Now read the titles of each review and start going through the ones that seem applicable to you.

  • Note that they are sorted with the most recent first, which is helpful as newer reviews tend to be more comprehensive.
  • 3ie has made the hardest part – assessing how strong the evidence is – easy for us. They use a 3-star scale to indicate the level of confidence in the systematic review.
  • In this example, the most recent review, Interventions For Improving Coverage Of Childhood Immunisation In Low- And Middle-Income Countries, is also the only one rated 3 stars (high quality). Click on its title.

The next page gives you an overview of the study. If it is “Open access”, this means you can read it for free – click “Go to source” below the star rating. If it isn’t open access, you can try some of the strategies in Step 9 to see if you can find the study for free elsewhere.

Clicking on “Go to source” opens a new tab with a PDF of the article. Don’t be intimidated by the length and technical terminology, and start with the summary – these articles usually include an “abstract” and sometimes a “plain language summary” and/or “summary of findings”.

The summary will likely be useful but too vague – dig into the review and look for details about which programs were tried and where, and how well they worked.

  • Keep track of which programs and studies seem particularly relevant, so that you can look them up later.
  • Consider the conclusions of the authors of the systematic review – are there trends that emerge across countries and contexts that are relevant for you? Overall, what are the main lessons from this review that you take away?
  • Be sure to read with a skeptical mindset – just because something worked in Japan doesn’t mean it will work in India – nor does it mean it won’t. And just because something worked in India before doesn’t mean it will continue to work – context is more complicated than country! Think about the evidence you find as well-documented ideas, but not the last word.
  • You can skip the parts about the methodology followed by the authors of the systematic review.
  • If the systematic review was helpful, add it to the bibliography.
  • Copy the citation from the references section and paste it into a search engine.
  • Usually, the first result will be the full paper on a journal’s website. If it is open access, you can read it directly. A lot of academic literature is not open access, unfortunately. Here are some tricks you can try if the article is not available:

a) Go back to your search and see if you can find a PDF posted on one of the authors’ websites – authors often share “working papers”, which might differ only slightly from the final paper, for free on their site.

b) Email the paper’s authors if you can’t find it elsewhere – many researchers are happy to share a copy with people looking to learn from their experience.

  • Read the paper. You probably don’t need to read all of it – the abstract, introduction, a description of the program, the results, and the conclusion are probably enough, whereas sections on technical methods can be skipped. This paper may also include references to other literature that could be relevant to your program. Keep track of these references so you can look into them later.
  • Add the paper to the annotated bibliography if it seems relevant.
  •  

For each piece of evidence that you find, there should be a clear justification for including it in the bibliography, such as: it is a landmark study in the topic (i.e. it has a large number of citations or is cited by many other studies in your review), it is relevant to specific aspects of this evaluation (such as measuring similar outcomes, being conducted in a similar context, or evaluating a similar intervention), etc. However, there are no absolute standards for inclusion, and since not all studies will be used in writing up the review, it is better to err on the side of including a study in the annotated bibliography.

Repeat step 9 for every paper from the systematic review that you found relevant!

  • If an organization ran the program that was evaluated, you might be able to find information on the organization’s website. If not, try emailing the organization.
  • Sometimes people share program details elsewhere – blogs, policy briefs, videos, etc. Try searching more about the program and you might find something.
  • In our example, we found a review from 2016, which is quite recent. However, keep in mind it can take 1-2 years to write and publish a review – and since the review is citing only published literature, the cited articles would be a year or two old as well. This means that it is likely that this review is missing anything done since 2014 or 2013, and hopefully the world has learned a lot about how to address your problem since then. While this shortcut helped you find relevant evidence, you might still want to use Method 2 so that you can see what has been learned since 2014.

 

  • This method used only academic sources. Non-academic sources are also a very useful source of information, and you should look into them. In Method 2, we have included a list of non-academic sources to consult.

Process evaluation vs monitoring: Which one do you need?

Process evaluations and monitoring both provide information on how your program is running and whether it is meeting expectations. The key difference is that process evaluations are a one-off activity, while monitoring is ongoing. That means that process evaluations are often more intensive exercises to collect more data and dive deeper into the theory of change. In contrast, ongoing monitoring must not overburden program staff and often tracks just a few high-priority indicators that are critical to program success.  

Consider the following questions to help you decide between conducting a process evaluation and building a monitoring system:  

1.     Are you interested in identifying specific problems or general problems? 

Process evaluations typically identify general or systemic problems along the theory of change, whereas monitoring typically identifies specific entities (e.g. service providers or locations)that need more attention.    

2.     Are you looking to hold program staff accountable?   

Both process evaluations and monitoring are implemented for learning – is our program being implemented as planned? If not, at which steps is it breaking down? However, if you are seeking an accountability system, monitoring is better-suited as it is continuous, whereas a process evaluation is a one-time exercise.  

3.     Do you need ongoing data on how your program is performing?  

A process evaluation typically offers a snapshot in time, whereas monitoring involves ongoing data collection and analysis for the entire duration of the program. For example, a process evaluation may do in-depth interviews with program participants on their experiences, whereas a monitoring system might collect data on just a few questions related to beneficiary satisfaction.  

4.     Do you need comprehensive data?

‍A process evaluation is typically based on a sample, while monitoring is usually comprehensive. For example, in a teacher training program, you would monitor the training of all teachers (because it is useful to know exactly which teachers did not attend the training), whereas in a process evaluation, you would interview a subset of teachers to understand the reasons why they did not attend the training.