IDi-Logo-Master-RGB-White-1-1.png

Impact Measurement Guide Beta

About monitoring
Example
Conducting monitoring
Case study ->

Monitoring

A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. It helps you identify which parts of implementation are going well and which are not.

Icon_Evie-01-pg57cdsb6m2opsz6e9pafm3yqw7kyn2k40yq2374hc-1.jpg
Click here to find out how its different

Example

You can use monitoring data to manage performance and learn where your program is not working as expected. For example, if you are running a teacher training program, you could track which teachers did not attend the training and whether teachers who attended implemented what they learned. You can then quickly identify which goals are not being met, and you would know how to investigate why. For example, if you find that only a small percentage of teachers are using the material, you can conduct follow-up interviews to understand the underlying reasons. This information can then inform improvements to your program.

Which questions can a needs assessment answer?

Tracking implementation will help you identify which areas of your program should be strengthened. For example, consider a program that provides information to farmers through phone calls that play pre-recorded material. Tracking whether people are answering the calls and listening to the full message can help you identify problems early on. You may learn that your phone numbers are out of date or that the numbers do not belong to your target audience (e.g. you may be reaching migrant workers in cities rather than rural farmers).

Generating evidence for decision-making is just one use of monitoring –a monitoring system is often used for other functions such as time reporting, payroll, and staffing. Donors, in particular, may expect monitoring data for accountability, to ensure you are making good use of program funds.

Next steps - Conducting monitoring

Monitoring requires that your organization collect, store, analyze, and visualize data on an ongoing basis. The data should be presented in a format that is useful for decision-makers. Using this data should be an established part of organizational decision-making.

Should you set up a monitoring system yourself or hire an expert?

Ultimately, your organization will need to conduct monitoring on its own – so your system needs to be set up in a way that program staff can use it independently. However, setting up the monitoring system can be done in-house or be outsourced.

Monitoring should be closely integrated with implementation. Since you are the best expert on what your program is trying to achieve and how it works, you should plan to be closely involved in setting up the monitoring system. You may want to hire an external expert if you do not have the technical capacity to set up a system for collecting, storing and presenting data in useful ways. It can also be useful to get an expert’s feedback on ways to measure the indicators you need.

How to conduct monitoring

Our step-by-step instructions and template will help you develop a monitoring system suited to your program needs.

Use the Theory of Change Builder to create a clear narrative for why your program will work.
Already created a theory of change? Log in and make sure it is up to date! We’ll refer to it going forward.
You can still proceed without linking your theory of change, if you want to preview the steps or have a theory of change saved elsewhere.

An implementation plan is more detailed than a theory of change, and includes the details of how you’ll put your program into effect – who will do what with which resources and materials. 

You can add implementation nodes to your theory of change, if you like.

Icon_Evie-01-pg57cdsb6m2opsz6e9pafm3yqw7kyn2k40yq2374hc-1.jpg

An implementation plan provides the operational details of how a program will run – who will do what with which resources. It also includes when, where, and how frequently these activities will happen. Your implementation plan should also include benchmarks for program success, such as targets for program uptake, service delivery, or program participant satisfaction.  

When an implementation plan is not written down in detail, there may be a lot of implicit (and potentially conflicting) knowledge about what should be happening. It is important to document program details so you can compare what’s happening on the ground to what was expected.

These may be shaky assumptions that you want to test, potential performance issues you should check for, or expected results that are closely tied to your impact. For example, say you are running a program in which health workers go door-to-door to remind parents to take their children to local vaccination camps. Some steps you may want to monitor are: 

  • Do health workers receive timely information on when vaccination camps will be held in the health center? 
  • Are health workers reaching their weekly targets for number of home visits?

You can identify shaky steps by asking: 

1. How critical is this step for program success? 

2. How certain are we that this assumption will hold in practice? 

The Theory of Change Builder automates this process for you! Log in to access a saved theory of change, and we’ll guide you towards identifying shaky steps. If you’re working with an offline theory of change, you can follow these instructions to identify shaky steps.

The indicators you choose should meaningfully answer your questions and be easy to measure reliably. Most of the time, you can identify indicators already in use rather than inventing your own. 

For example, here are some sample indicators for the health worker home visit program: 

While the Impact Measurement Guide focuses on data to measure and improve program performance, your monitoring system can also help you collect data for other purposes, such as payroll, staffing and donor reporting requirements. For example, some donors may ask for demographic information about people participating in the program. 

You can use the indicator recommender to shortlist metrics!

Create a response framework that describes whether and how your organization will act on the data you collect. Often, this means setting thresholds for indicators, and defining meaningful actions your organization will take if these thresholds are crossed. As an example, have a look at this sample response framework for the three indicators we created for the heath worker home visit program. 

A good rule of thumb is: if you can’t think of an action your data would inform, you may not need that indicator!  

Most nonprofits will use administrative data that is already available to them, either collected on their own or by government agencies. The data collected at registration or data measuring program activity can prove to be valuable for the purpose of monitoring and observing whether program targets are being met.

To identify which data collection method would suit your needs, you need to know: Who or what do you need data on? What is the best way to obtain it? How often does data need to be collected? 

In some cases where new data needs to be collected for monitoring, surveys can be conducted. For example, to gauge the level of satisfaction of the parents bringing their children to vaccination centres, satisfaction surveys can be conducted while vaccinations centres are running.

You also need to consider how to record the data you are collecting. Typically, organizations conduct surveys in-person or over the phone. You can set up a simple data collection system that use pen-and-paper surveys, direct data entry into spreadsheets, or digital surveys. Click here for more guidance on developing a data collection system. 

Icon_Evie-01-pg57cdsb6m2opsz6e9pafm3yqw7kyn2k40yq2374hc-1.jpg
In some cases, you may be able to use data collected by external organizations, such as web traffic data collected by Google Analytics. If you’re planning on using any third-party data, make sure you understand how the data was generated, how data quality checks and cleaning are carried out, and how aggregations are calculated. IPA provides extensive guidance on how to use administrative data for monitoring and evaluation.

Where will the data you collect go? How will it be stored? For basic monitoring systems, data can be stored as data files (.xlsx, .csv etc.) in the cloud using Dropbox or Google Drive. This ensures that data can easily be accessed using commonly-used spreadsheet software (such as Microsoft Excel or Google Sheets) – and it also backs up your data. As you set up these technologies, also remember that you also need to train staff to maintain and operate these systems. 

As your organizations grows and your needs become more complex, you may decide to invest in more expensive and/or customized systems, such as an SQL database server. 

Icon_Evie-01-pg57cdsb6m2opsz6e9pafm3yqw7kyn2k40yq2374hc-1.jpg

It’s important to develop strong data security practices that protect the identity and privacy of your respondents, particularly if you are collecting confidential or sensitive information. Paper survey responses should be stored securely (e.g. in a locked cabinet), and electronic data should be protected with strong passwords. You can also consider encrypting your data.

Preparing the data for analysis is called data cleaning. A large part of this process involves addressing data entry errors. This should be automated as much as possible in the previous steps. For example, if you are asking respondents their phone number in a digital survey, you can set a constraint that the response be exactly ten digits long. 

A monitoring system is useful only if it is regularly informing organizational decision-making – so make sure your response framework is integrated into the monitoring system! Does the information from your monitoring system reach the right decision-makers? Does this information arrive in time and in a format they can use easily? Does this information translate into meaningful actions to improve the program? 

Congratulations! If you’ve come this far, you’ve set up a monitoring system that is giving you regular, actionable information about how your program is performing. Now, your challenge is to ensure your monitoring system continues to be responsive to your program’s needs. As you resolve certain problems and shift your focus to others, you will need to revise the indicators your monitoring system collects. Periodically review whether your monitoring system is giving you all the answers you need, and whether you are using all the data your monitoring system is collecting.

You can use this checklist to assess your monitoring system and identify areas of improvement. 

Set an email reminder to review and update your monitoring system.

Guide

Not sure where to go from here? 
Use our guide to frame a question and match it to the right method.

Monitoring case study

How a monitoring system enabled the Delhi government to develop a data-informed policy response to COVID-19

IDi-Logo-Master-RGB-White-1-1.png

Impact Measurement Guide

Twitter Linkedin Instagram Facebook

Feedback

The Impact Measurement Guide is still in beta release, we welcome your feedback on which features you like as well as what we should change or improve in future versions.

Feedback Form ->

Information

  • About
  • Terms & Conditions
  • Privacy Policy

Contact us

  • guide@idinsight.org
  • SALE ENDS IN:
Days :
Hours :
Minutes :
Seconds

— NEXT ONLINE COURSE STARTS ON MAY 26TH —

UI/UX Fundamentals
25% OFF

Enroll Now

Types of impact evaluations

Impact evaluations compare the people who receive your program to a similar group of people who did not receive your program. Based on how this similar group is chosen, impact evaluations can be randomized controlled trials or quasi-experimental.

A randomized controlled trial (RCT) is considered the gold standard of impact evaluation. In an RCT, the program is randomly assigned among a target population. Those who receive the program are called the treatment group; those who do not receive the program are called the control group. We consider the outcomes from the control group to represent what would have happened to the treatment group in the absence of the program. By comparing outcomes among those who receive the program (the treatment group) to those who don’t (the control group), we can rigorously estimate the impact of the program. The random assignment of the program to the treatment and control group provides the rigor, as it ensures that the selection of people is not based on biased criteria that could affect the results.

When a randomized design is not feasible, there are other, “quasi-experimental,” ways of constructing a valid comparison group. 

  • In a matched design we would match individuals who receive the program to individuals who don’t receive the program based on some observable characteristics (such as age, gender, number of years of schooling, etc.), and compare outcomes across these groups. 
  • Another common technique is regression discontinuity design, in which you create a cutoff based on which individuals are eligible to receive the program, and then compare outcomes from groups just below and just above the cutoff to estimate impact. 

Matched designs and regression discontinuity designs are just two of many quasi-experimental techniques. J-PAL provides an overview of common methods of conducting an impact evaluation. All such methods seek to identify what would have happened to your target population if they had never received the program, and their success relies on the strength of the assumptions they make about whether the comparison group is a credible stand-in for your program’s target population. 

Recommendation: Theory of Change

A theory of change is a narrative about how and why a program will lead to social impact. Every development program rests on a theory of change – it’s a crucial first step that helps you remain focused on impact and plan your program better.

You can use diagrams.net to create your Theory of Change.

Learn more about the theory of change ->
Go to the next state of the decision guide

Recommendation: Needs assessment

A needs assessment describes the context in which your program will operate (or is already operating). It can help you understand the scope and urgency of the problems you identified in the theory of change. It can also help you identify the specific communities that can benefit from your program and how you can reach them.

Build your theory of change ->

Once you’re satisfied that your program can be implemented as expected:

Go to the next state of the decision guide

Your program's theory of change

You have a written plan for how your program will improve lives – great! Make sure to refer to it as you explore the different sections in the Impact Measurement Guide, as it is the foundation for any other method you’ll use.

Learn more about the theory of change ->
Go to the next state of the decision guide

Recommendation: Process Evaluation

A process evaluation can tell you whether your program is being implemented as expected, and if assumptions in your theory of change hold. It is an in-depth, one-time exercise that can help identify gaps in your program.

Learn about process evaluations ->

Once you are satisfied that your program can be implemented as expected:

Go to the next state of the decision guide

Recommendation: Evidence Review

An evidence review summarizes findings from research related to your program. It can help you make informed decisions about what’s likely to work in your context, and can provide ideas for program features.

Learn about evidence reviews ->

Once you are satisfied with your evidence review:

Go to the next state of the decision guide

Recommendation: Monitoring

A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.

Learn more about monitoring ->

Once you are satisfied with your monitoring system:

Go to the next state of the decision guide

Recommendation: Monitoring

A monitoring system provides continuous real-time information about how your program is being implemented and how you’re progressing toward your goals. Once you set up a monitoring system, you would receive regular information on program implementation to track how your program is performing on specific indicators.

Learn more about monitoring ->

Once you are satisfied with your monitoring system:

Go to the next state of the decision guide

How to compile evidence, method 2

You’re on this page because you want to search for evidence relevant to your program.

Here are some academic sources where you can search for relevant research:

  • Google Scholar is a search engine for academic papers – enter the keywords relevant to your program, and you’ll find useful papers in the top links
  • Once you identify some useful papers, you can consult their literature review and bibliography sections to find other papers that might be relevant
  • Speaking to a sector expert can guide you to useful literature

 However, don’t include only academic studies in your review! You should also consult:

  • Policy reports
  • Websites of organizations involved in this issue, such as think tanks, NGOs, or the World Bank
  • Public datasets
  • Your program archives – data and reports from earlier iterations of the program can be very valuable!

 The free Zotero plug-in provides an easy way to save, organize, and format citations collected during internet research. Note that Zotero can help you start your annotated bibliography, but it is not a substitute since it does not include any summary or interpretation of each study’s findings.

How to compile evidence, method 1

Start with the 3ie Development Evidence Portal, which has compiled over 3,700 evaluations and over 700 systematic evidence reviews. Steps 2-7 of this example are specific to locating evidence on the 3ie website, but you can also consider looking for a review by J-PAL or Campbell Collaborations or the Cochrane.

For example, suppose your goal is to increase immunization rates in India. Type “immunization vaccination” or other related terms into the search box, and click the magnification lens to search.

The search results include individual studies, which are usually about a single program in a single location, as well as “systematic reviews”, which is what we are looking for because they are more comprehensive. To show only the systematic reviews, on the left of the screen under Filter Results, click on PRODUCTS and check the Systematic Reviews box. We’re now left with 17 evidence reviews related to immunization.

Now you might want to further narrow your search by region or country. In our example, suppose we want to see only those evidence reviews that contain at least one study from India. Click on COUNTRY and scroll down to click on India.

There are still 9 evidence reviews! Now read the titles of each review and start going through the ones that seem applicable to you.

  • Note that they are sorted with the most recent first, which is helpful as newer reviews tend to be more comprehensive.
  • 3ie has made the hardest part – assessing how strong the evidence is – easy for us. They use a 3-star scale to indicate the level of confidence in the systematic review.
  • In this example, the most recent review, Interventions For Improving Coverage Of Childhood Immunisation In Low- And Middle-Income Countries, is also the only one rated 3 stars (high quality). Click on its title.

The next page gives you an overview of the study. If it is “Open access”, this means you can read it for free – click “Go to source” below the star rating. If it isn’t open access, you can try some of the strategies in Step 9 to see if you can find the study for free elsewhere.

Clicking on “Go to source” opens a new tab with a PDF of the article. Don’t be intimidated by the length and technical terminology, and start with the summary – these articles usually include an “abstract” and sometimes a “plain language summary” and/or “summary of findings”.

The summary will likely be useful but too vague – dig into the review and look for details about which programs were tried and where, and how well they worked.

  • Keep track of which programs and studies seem particularly relevant, so that you can look them up later.
  • Consider the conclusions of the authors of the systematic review – are there trends that emerge across countries and contexts that are relevant for you? Overall, what are the main lessons from this review that you take away?
  • Be sure to read with a skeptical mindset – just because something worked in Japan doesn’t mean it will work in India – nor does it mean it won’t. And just because something worked in India before doesn’t mean it will continue to work – context is more complicated than country! Think about the evidence you find as well-documented ideas, but not the last word.
  • You can skip the parts about the methodology followed by the authors of the systematic review.
  • If the systematic review was helpful, add it to the bibliography.
  • Copy the citation from the references section and paste it into a search engine.
  • Usually, the first result will be the full paper on a journal’s website. If it is open access, you can read it directly. A lot of academic literature is not open access, unfortunately. Here are some tricks you can try if the article is not available:

a) Go back to your search and see if you can find a PDF posted on one of the authors’ websites – authors often share “working papers”, which might differ only slightly from the final paper, for free on their site.

b) Email the paper’s authors if you can’t find it elsewhere – many researchers are happy to share a copy with people looking to learn from their experience.

  • Read the paper. You probably don’t need to read all of it – the abstract, introduction, a description of the program, the results, and the conclusion are probably enough, whereas sections on technical methods can be skipped. This paper may also include references to other literature that could be relevant to your program. Keep track of these references so you can look into them later.
  • Add the paper to the annotated bibliography if it seems relevant.
  •  

For each piece of evidence that you find, there should be a clear justification for including it in the bibliography, such as: it is a landmark study in the topic (i.e. it has a large number of citations or is cited by many other studies in your review), it is relevant to specific aspects of this evaluation (such as measuring similar outcomes, being conducted in a similar context, or evaluating a similar intervention), etc. However, there are no absolute standards for inclusion, and since not all studies will be used in writing up the review, it is better to err on the side of including a study in the annotated bibliography.

Repeat step 9 for every paper from the systematic review that you found relevant!

  • If an organization ran the program that was evaluated, you might be able to find information on the organization’s website. If not, try emailing the organization.
  • Sometimes people share program details elsewhere – blogs, policy briefs, videos, etc. Try searching more about the program and you might find something.
  • In our example, we found a review from 2016, which is quite recent. However, keep in mind it can take 1-2 years to write and publish a review – and since the review is citing only published literature, the cited articles would be a year or two old as well. This means that it is likely that this review is missing anything done since 2014 or 2013, and hopefully the world has learned a lot about how to address your problem since then. While this shortcut helped you find relevant evidence, you might still want to use Method 2 so that you can see what has been learned since 2014.

 

  • This method used only academic sources. Non-academic sources are also a very useful source of information, and you should look into them. In Method 2, we have included a list of non-academic sources to consult.

Process evaluation vs monitoring: Which one do you need?

Process evaluations and monitoring both provide information on how your program is running and whether it is meeting expectations. The key difference is that process evaluations are a one-off activity, while monitoring is ongoing. That means that process evaluations are often more intensive exercises to collect more data and dive deeper into the theory of change. In contrast, ongoing monitoring must not overburden program staff and often tracks just a few high-priority indicators that are critical to program success.  

Consider the following questions to help you decide between conducting a process evaluation and building a monitoring system:  

1.     Are you interested in identifying specific problems or general problems? 

Process evaluations typically identify general or systemic problems along the theory of change, whereas monitoring typically identifies specific entities (e.g. service providers or locations)that need more attention.    

2.     Are you looking to hold program staff accountable?   

Both process evaluations and monitoring are implemented for learning – is our program being implemented as planned? If not, at which steps is it breaking down? However, if you are seeking an accountability system, monitoring is better-suited as it is continuous, whereas a process evaluation is a one-time exercise.  

3.     Do you need ongoing data on how your program is performing?  

A process evaluation typically offers a snapshot in time, whereas monitoring involves ongoing data collection and analysis for the entire duration of the program. For example, a process evaluation may do in-depth interviews with program participants on their experiences, whereas a monitoring system might collect data on just a few questions related to beneficiary satisfaction.  

4.     Do you need comprehensive data?

‍A process evaluation is typically based on a sample, while monitoring is usually comprehensive. For example, in a teacher training program, you would monitor the training of all teachers (because it is useful to know exactly which teachers did not attend the training), whereas in a process evaluation, you would interview a subset of teachers to understand the reasons why they did not attend the training.