Process evaluations
A process evaluation tells you whether your program is being implemented as expected. In other words: process evaluations help you see whether the steps and assumptions in your theory of change holdup in practice.
Example
A process evaluation can help you:
- Track whether the iron supplements you purchased are reaching the people they’re meant for
- Assess whether farmers are remembering the lessons from their extension trainings
- Learn whether students are using the computers that you distributed for classroom instruction
Which questions can a process evaluation answer?
Process evaluations often identify elements of program design that should be modified based on gaps observed in the field. For example, you may run a program that trains veterinary health workers to provide services such as vaccination and deworming to all livestock farmers. A process evaluation may tell you that veterinary health workers are only selling to medium- and large-sized livestock farmers. As a result, you may consider adding monetary incentives for reaching smallholders. Similarly, you may want smallholder farmers to make up at least 50% of the health workers’ clientele, but your process evaluation reveals that it is only 10%. Therefore, it helped you recognize that your program was falling short of a target during implementation, and you may now take up steps to rectify that.
A process evaluation may reveal that it is not possible to implement the program in accordance with the theory of change. For example, if an assumption critical to the success of your program does not hold, and is outside of your control, you may need to consider terminating the program. Say you have a program that offers doorstep banking services for women. Banking agents take deposits from customers and drop the money they collect at the bank daily. However, when you conduct a process evaluation, you learn that the roads have broken down after a recent storm, and rebuilding will take a while. It is not safe for agents to travel on these roads. In this case, you would need to consider terminating the program (or significantly changing how it is set up).
Process evaluations often point out areas for improved monitoring and oversight. For example, if a process evaluation for a program providing extension training to farmers reveals that you are not reaching smallholder farmers, you may want to start regularly collecting customer profile data to track program reach.
Next steps - Conducting a process evaluation
Doing a process evaluation involves two steps:
- Assessing program implementation: Data on implementation is collected through activities such as direct observation of program activities, surveys of staff and participants, focus groups with participants, audits of documentation or monitoring data, etc.
- Comparing these process descriptions to expectations: This determines whether what is happening on the ground is in line with the path to impact outlined in the theory of change.
Should you do the process evaluation yourself or hire an expert?
Organizations with the capacity to run surveys, conduct interviews, and analyze data can do process evaluations on their own; however, you may want to hire an external party to reduce the risk of bias.
If you are conducting a process evaluation to meet high-stakes funder requirements, the evaluation would be more credible if it were conducted by an external evaluator, as the risk of bias is lower. If, however, you are conducting a process evaluation for internal learning, you can do the exercise in-house if you have capacity and are confident that you can collect unbiased data.
To run a process evaluation in-house, your organization should be able to collect quality data reliably. Your team will need to design questionnaires and responsibly survey implementers and/or participants on a variety of topics. We generally recommend hiring surveyors externally so as to minimize response bias.
Your organization should also be able to analyze qualitative and quantitative data. Qualitative data analysis requires the ability to organize high volumes of information and the ability to read interview answers with a careful eye. Quantitative data analysis may require only simple summary statistics in a spreadsheet program like Microsoft Excel, or more nuanced econometric analyses using specialized software like Stata or R.
If you do not have internal capacity to reliably collect, analyze, and interpret the data, you should work with an external party.
If you choose to conduct the process evaluation externally, you’ll save on staff time. Note that someone from your organization who is familiar with the program will still need to spend time working closely with the evaluator and supporting them in data collection.
How to conduct your process evaluation
Our step-by-step instructions and template will help you design and execute a process evaluation
Use the Theory of Change Builder to create a clear narrative for why your program will work.
Already created a theory of change? Log in and make sure it is up to date! We’ll refer to it going forward.
You can still proceed without linking your theory of change, if you want to preview the steps or have a theory of change saved elsewhere.
An implementation plan provides the operational details of how a program will run – who will do what with which resources. It also includes when, where, and how frequently these activities will happen. Your implementation plan should also include benchmarks for program success, such as targets for program uptake, service delivery, or program participant satisfaction.
When an implementation plan is not written down in detail, there may be a lot of implicit (and potentially conflicting) knowledge about what should be happening. It is important to document program details so you can compare what’s happening on the ground to what was expected.
An implementation plan is more detailed than a theory of change, and includes the details of how you’ll put your program into effect – who will do what with which resources and materials.
You can add implementation nodes to your theory of change, if you like.
Highlight any important nodes or links you’re unsure of. If your theory of change says that teachers will attend trainings, learn better teaching practices, and apply them in their own classrooms, ask yourself: How will I know if teachers come to my trainings? If they understand the curriculum? If they plan to use the practices we teach them?
You can identify shaky steps by asking:
- How critical is this step for program success?
- How certain are we that this assumption will hold in practice?
The Theory of Change Builder automates this process for you! Log in to access a saved theory of change, and we’ll guide you towards identifying shaky steps. If you’re working with an offline theory of change, you can follow these instructions to identify shaky steps. We also provide a template to track shaky steps.
Process evaluations can use quantitative data (anything that can be represented as a number) or qualitative data (harder-to-summarize observations and conversations). Good process evaluations often combine both.
Choose metrics that meaningfully capture the things you care about. For example, a crucial – and uncertain – link in your theory of change might be whether teachers understand and act on the training you provide. You could ask teachers point-blank whether they plan to apply your training to the classroom, but you would get richer and more accurate information by observing them teaching or asking their students.
You can use the indicator recommender to shortlist metrics!
Decide what levels for each metric indicate success or failure. Quantify as much as possible. For example, saying that the program is successful if you see teachers implementing applying what they’ve been trained on in their classrooms is ambiguous. How many teachers need to be acting on the training? How many of the new teaching practices should they be applying? Are “all” practices necessary, or would half of the practices be sufficient to meet our definition of success?
Choose a data collection platform and assemble a team of enumerators. It’s often cheaper to have program staff collect data, but that can be tricky if there any conflicts of interest. If independence is important, consider hiring outside surveyors.
Data analysis for process evaluations is fairly straightforward, since you’re summarizing your data to compare with your expectations for the program, rather than a control group. Analyzing quantitative data can often be done in Excel or Google Sheets. Analyzing qualitative data can be more time-consuming and can take a range of approaches.
Finding weaknesses in your program is good news – it gives you a chance to fix them before investing in an impact evaluation or large-scale growth. If you find that your program is falling short in specific areas, go back to the drawing board! Depending on your findings, you may need to talk more with your target users about their needs; explore how other programs have handled the problem; or run some quick tests of variations of your program.
Guide
Not sure where to go from here? Use our guide to frame a question and match it to the right method.
Process evaluation case study
How a process evaluation for Farm to Market Alliance in Zambia identified changes that could make the program more effective
