A guide to monitoring and evaluation for NGOs
Monitoring and evaluation (M&E) is a methodical process of collating and assessing information to determine the effectiveness of a social programme or intervention. This website is an M&E resource for non-profit organisations (NPOs) and public benefit organisations (PBOs). Those with little M&E experience will be able to use the step-by-step guides to plan, monitor and evaluate programmes and interventions. More experienced users can access extensive resources to improve existing M&E processes or simply learn from other practitioners and organisations.
Find info
Plan for effective monitoring and evaluation of your programme
A Monitoring and Evaluation (M&E) plan is important to determine if a programme is effective and defines what needs to be assessed and which information is important to gather. M&E plans are best developed during the design phase of the programme as good evaluations rely on strong data, systematically collected. Don’t be anxious if you do not have an M&E plan yet; just develop one that works for you as soon as possible. The adjacent steps have been set out in a logical order to help you do this, but feel free to jump around and look at different aspects of M&E planning in the way that works for you.
Step 1: What is Your Theory of Change?
How can a programme or intervention create change in a situation or community? A Theory of Change describes the most important processes, drivers or mechanisms needed to create change. The development of a Theory of Change can be based on research information, complemented by an intuitive and experiential understanding of how things work.
Why is a Theory of Change Important?
A Theory of Change examines our ideas or assumptions of how change is expected to happen. It can be very difficult to interpret the results of an evaluation without any programme theory, as we will illustrate. Let’s use an example of a hypothetical programme that tries to improve the nutrition of teachers in an underprivileged community, by teaching them to cultivate tomato plants in school gardens. How does this work? This is precisely what your Theory of Change explains.
At the outset, this programme invests resources such as seed, soil, water and fertilizer. Through different activities, like training teachers to plant, fertilize and water the tomato seeds, the programme staff hope that the teachers will soon be producing tomato plants. One of the products of the programme is tomato plants; another is teachers who are trained to cultivate the tomato plants. We call these programme outputs (see the infographic on M&E planning terminology).
A short-term outcome is a reasonable harvest of ripe, succulent tomatoes.
The theory of the programme is this. If the teachers eat more tomatoes, they will experience better nutrition and health in the long term. They also might be able to sell the surplus tomatoes and use the extra cash to buy more food and supplement their diets further. In a chain-effect the programme staff hope that teachers will ultimately become healthier and more effective teachers and from this provide better education to the children they teach.
Imagine if you are evaluating the programme and you assume that this programme will ultimately be effective if it shows that teachers who have participated in the programme are healthier than before they took part. This assumption would, of course, be correct. You therefore don’t give a lot of attention to how the programme is supposed to work in theory, but choose to cut to the chase and assess the level of health of teachers before and after the programme. You find no difference in their health and so conclude that the programme is ineffective.
Technically, you have evaluated the programme, but it has not been a very helpful evaluation as it leaves the programme staff with nothing but a sense of failure. You will not have answered the key question – why is the programme ineffective? Is it because the idea is not viable? Or did the teachers struggle to look after the plants because of a lack of water or disease? Perhaps the tomatoes were not harvested when they should have been, or the teachers did not cultivate them properly due to a lack of motivation or understanding of the training. The list could go on.
If the time is taken to document a Theory of Change during programme design, then this can be used to assist people in choosing activities and identification and mitigation of risks at an early stage. For example, as part of the Theory of Change you will identify some of your underlying assumptions. In this example, one might be assuming that there is an adequate water supply to the school gardens. Noting this will ensure that the programme staff are aware that this is a key factor for success. This might result in staff developing site assessment procedures before they start implementing the programme at a particular location. If they design an M&E system to collect information on all the different aspects of their Theory of Change, their evaluations will not only tell them whether their programme was effective or not but also why.
Feel free to explore other interpretations of the Theory of Change. Actknowledge and the Theory of Change Community have developed a complete programme design and M&E planning process. You can access these resources and other free software from the Toolbox.
Terminology Used in M&E Planning
There are many frameworks and methods that you can use for programme design/M&E planning, each with specific terminology. It is helpful to know what these words mean to apply them correctly. For simplicity and clarity, terms are explained through the example of the programme planting and cultivating tomatoes to improve the nutrition of a community of school teachers (see ‘Why is the Theory of Change important?‘).
- Inputs are the resources that you need to implement your programme, such as staff expertise or infrastructure. In the tomato programme example, soil, seed, fertiliser and water are all required inputs as well as available workers and management.
- Activities describe what a programme has to do to bring about the anticipated change. Example activities in the tomato programme are the training of the teachers, preparation of the soil, planting the seeds and watering them for a successful harvest.
- Outputs are the products or services created and delivered by a programme. For example, planting tomato seeds, fertilising and watering the soil are activities likely to result in the growth of green plant shoots – direct results. The programme does not show improved nutriton and health indicators yet, but it is showing that the process to achieving the end result is on track.
- Outcomes describe the short-term and intermediate changes that occur as a direct result of programme activitities/products/services. For example, a harvest of healthy big tomatoes indicates something has been achieved (food source) which is one outcome.
- Impact implies the bigger, longer term changes in communities, organisations, society or the environment as a result of programme outcomes. If the example school teachers are able to eat the tomatoes and experience improved nutrition and health (or sell produce to supplement income and therefore diet) we can say the impact is evident.
Aims, goals and objectives
The tomato plant example is taken from the M&E Blog
How to Develop a Theory of Change
- Who will take the lead in the development of the Theory of Change and who will participate? This is the first decision that needs to be made. There is good reason to develop programme theory (see ‘Why a Theory of Change is important’ via the video or read the text). Who leads and who participates in the programme will depend on your reasons for developing one. The best programme theories are developed with group input and the process of doing so is beneficial to all the participants. To save time, a draft can be developed by one person and then reviewed in a group. For an evaluation plan, it is important to include all the programme managers in this process.
- If you are already implementing a programme, it is likely you have already done a situation analysis which has informed the programme strategy and activities. This is the starting point for the Theory of Change. If you are beginning to develop a programme, a situation analysis might include a detailed process of reviewing research and interviewing community members or stakeholders to gather data for analysis. If you have already done this, it is worthwhile providing a brief descriptive summary of the problem to give context to your Theory of Change. Answering focused questions can guide you in doing the situation analysis. For more information, see ‘Tool: Ways to do a Situation Analysis and to Describe Your Programme Focus‘ and read our case study for an example. Something else that might be useful is our ‘Quick Guide to Doing a Literature Review for a Situation Analysis.’
- The next step is focusing – deciding on and articulating your programme strategy. In doing this you will look at the limitations or boundaries of your programme. In the situation analysis, which parts of the problem or opportunity will you address and what will be out of programme scope or boundary? It is important to try and foresee if the parts that fall outside of the scope will impact on your intended outcomes. For example, an organisation providing skills training for unemployed people might improve their employability, but the lack of job opportunities in the market will affect the ultimate goal of the programme (to reduce unemployment). Such aspects or variables must be taken into account in programme evaluation. Have a look at the case study for this section.
- After working on your programme focus, develop the ‘outcomes chain’ (see an example). An outcomes chain shows the assumed cause and effect between immediate, intermediate and ultimate outcomes or impact. An outcomes chain is a tool to use when thinking about how the programme functions. When a new programme is designed, this becomes the primary rationale for choosing programme activities, resource allocation and selecting performance indicators. There are various ways to develop an outcomes chain and it is up to you to try different methods and see what works best. For help, see the guidelines in ‘Ways to Develop an Outcomes Chain Tool‘. Have a look at the case study discussion and example before you start working on your own outcomes chain.
Quick Guide: Doing a Literature Review for a Situation Analysis
The following steps will guide you in doing a literature review for a situation analysis, although this does require a certain level of skill. The guidelines below will help you with the minimum information you need to know to perform a literature review that is reliable. The more effort you invest in further research the better you will be in practice.
A: Search for the right information to inform research
Research articles typically report on research which, because its methodology is rooted in science, allows us to say or understand something with a high level of confidence in the truth/validity of what is being described. Research papers are published in journals, like the The Southern African Journal of Education for example. People who are not staff members or students registered with a tertiary education generally have to pay to access them. This is particularly true for papers published in international journals. Payment can be made online and you will get immediate access to the article with the option of downloading it. Prices range between R80-R500 per article, depending on the exchange rate. A quick way to search for articles is to use the Google Scholar search engine. This will allow you to search all relevant publications to see whether or not they are free.
Reports and learning briefs are often generated and made available by non profit organisations or other agencies implementing and coordinating social interventions. Reports and learning briefs are based on actual experience (by the author or organisation) and so are very valuable resources when doing a literature review for a situation analysis. You should be able to access reports through a normal Google search; remember to experiment with different search terms as sometimes different combinations of words will give better results. If you find an organisation that is doing relevant work, contact them and ask them about their experiences even if they don’t have publications available. Some useful places to look are:
News and other internet sources are extremely useful. Newspaper articles (print and online) will inform you of recent developments in the area that you are interested in and Wikipedia is an excellent quick reference to understand the basics of concepts or issues. The link for the relevant term will usually appear in a Google search.
B: Study the information
When searching for information, you tend to gather many pieces of research that can look relevant, but turn out to be otherwise. Take the time to do an initial scan through information and then disregard, select or keep what you need from a search. From this, it is helpful to highlight quotes in the information that will be useful to refer to or read again when writing the review. If you prefer to work with printed documents, create written notes or use highlighters to underline important quotes. When working electronically, you can create a Microsoft Word table (watch a video showing you how) by coping and pasting the quote (from the PDF file or internet), adding the name of the source document and assigning it to one or more categories.
Try to create broadly defined categories which helps you to ‘chunk’ information easily – in this way it is easier to arrange and order quotes. You can sort the category column alphabetically and arrange the quotes under a certain topic. This will allow you to see which areas you have covered sufficiently and which you still need to find more information on. An example is shown below.
| Quote | Category | Source |
| Indeed, the article takes as its starting point the observation that NPOs in South Africa and elsewhere are increasingly challenged to demonstrate relevance and results due to the relative scarcity of development funds. | Increased pressure to do M&E | Mueller-Hirth1 |
| In other words, there is a significant gap between the extensive debates and innovations surrounding evaluation, and actual M&E practices in the selected NPOs. | M&E is becoming too complex for many organisations | Mueller-Hirth |
| The difficulty for many NPOs is that these systems make little sense. Many respondents from NPOs stated that the requirements were a distraction from their real work, confusing, redundant or destructive. | M&E is becoming too complex for many organisations | Bornstein2 |
| Capacity is at the core of the division between those who can satisfy donors’ requirements through effort and creativity and those who cannot. | M&E capacity is often lacking | Bornstein |
| 1Mueller-Hirth, N. 2012. If You Don’t Count, You Don’t Count: Monitoring and Evaluation in South African NGOs. Development and Change 43(3): 649–670.2Bornstein, R. 2006. Systems of Accountability, Webs of Deceit? Monitoring and Evaluation in South African NGOs. Development, 2006, 49(2), (52–61). | ||
C: Write the Review
(i) A logical flow and structure is really important.
Start by thinking through and drawing a flow chart of how you will structure the review. Always put yourself in the shoes of the reader. Firstly, the reader needs to be introduced to the topic. What is it about and why is it important? Next, systematically unpack and analyse the core issues and follow these with recommendations for the development of your programme or intervention after considering the issues. Finally, pull the main findings and recommendations together in a short conclusion. An example of a flow chart is given below for a literature review dealing with the topic of youth unemployment.
(i) Simple, clear, readable text is key.
Our top four writing guidelines are:
- Keep sentences and paragraphs short. Re-read sentences and cut them in two or three where necessary. This also applies to paragraphs.
- Write clearly and simply. In a country where most people use English as a second language, accessible and clear writing is particularly important.
- The spelling and grammar check feature of your word processing software is essential and should be used throughout work. Remember that good writing is influenced by the time spent crafting and arranging words in the right way.
- Edit, edit, edit. Read, re-read and change. Do this multiple times to refine your writing. You may need to ask someone else to read through your work for an unbiased and objective opinion as too much time spent in front of text can cause the best writer to miss things.
D: Reference
You need to reference the work of others in your text and provide a list of references at the end of your document. References for academic documents in the social sciences are usually based on American Psychological Association (APA) standards and guidelines. If you would like to follow these guidelines, the Purdue Online Writing Lab provides them. We find it easiest to reference using footnotes, particularly for copying and pasting information between documents as the sentence and footnote are copied together (see the video showing you how).
You must reference when describing the ideas and concepts of others or quoting text directly. You can either work the reference into the sentence with the year of publication in brackets such as: “Bornstein (2006) argues…; According to Bornstein (2006)…; Bornstein (2006) emphasises…”; or you can paraphrase or “quote” and simply put the surname of the author and the publication date in brackets at the end of the sentence (such as: Bornstein, 2006). Remember to provide the full reference as a footnote[1] next to the in-text reference or to provide the number of the footnote if it has already been given earlier. The following is the format of the four types of references that you will use most often:
| Type | Example | Formula |
A journal article: | Appleton, J. 2000. ‘At my age I should be sitting under that tree’: the impact of AIDS on Tanzanian lakeshore communities. Gender and Development, 8(2): 19-27. | Surname (s), Initials/Name of author organisaton. Publication year. Name of article. Name of journal in italics, Issue number (volume number in brackets): page numbers. |
| A book or report: | Altman, M and Marock, C. 2008. Identifying appropriate interventions to support the transition from schooling to the workplace. Pretoria: Human Sciences Research Council and Centre for Poverty Employment and Growth. | Surname (s), Initials/Name of author organisaton. Publication year. Name of report or book in italics. Place of publication: Name of publisher |
| An internet resource: | SASSA. 2011. You and your grants: 2011/12. Pretoria: SASSA (accessed from: http://www.sassa.gov.za/Portals/1/Documents/d54e383b-7e3d-4c96-8aa2-4cc7d32bc78f.pdf on 26 May 2012). | Surname (s), Initials/Name of author organisaton. Publication year. Name of document in italics. Place of publication: Name of publisher (accessed from: website address on [date accesed]) |
| A newspaper article: | Mail & Guardian. 2009. ‘Child support grant could extend to 18-year-olds’. The Mail & Guardian, 11 February (accessed from: http://mg.co.za/article/2009-02-11-child-support-grant-could-extend-to-18-year-olds on 26 May 2012). | Surname (s), Initials of journalist (if known)/Name of newspaper. Publication year. Name of article. Name Newspaper, date that article was published (accessed from: website addres on [date accesed]) |
[1] Bornstein, R. 2006. Systems of Accountability, Webs of Deceit? Monitoring and Evaluation in South African NGOs. Development, 2006, 49(2), (52–61).
Tools
Ways To Do Situation Analysis and Describe Your Programme Focus
Asking the why’s
- Start by stating the problem that you would like to address
- Answer a ‘why’ chain of questions. Start off by querying the problem – why does this happen?
- Ask another ‘why’ to your previous answer to continue interrogating the cause.
- Continue until the problem has been thoroughly explored
Now describe your programme’s focus:
- Which of the ‘why’ questions are crucial to address?
- Which of the ‘why’ questions will or does your programme address?
- Are there any other programmes that are addressing or will address the outstanding ‘why’ questions?
- What will is the effect or will be the effect of not addressing certain ‘why’ questions?
OR
Answering specific questions (source: Purposeful Program Theory: Funnell & Rogers, 2011)
- What is the problem, issue or opportunity?
- For whom does this programme exist?
- What is the history of the problem, and what projections are there about its future?
- Why does this problem occur? What are its causes? Are some causes more important or influential than others? Are there known casual pathways? (succession of causes)
- What if anything is known about what has and has not been effective in addressing this problem?
- Why should this be considered a problem? What are the consequences of this problem for those directly affected by it?
Describe the programme’s focus by:
- Identifying the main strategies that the programme uses or will use.
- Determining the desired outcomes and conditions that are to be addressed by the programme
- Determine what will happen for outcomes and conditions that fall outside the boundary of the programme focus.
Ways to Develop an Outcomes Chain
Listing and ordering outcomes | Working up: strategy to Impact | Working down: impact to immediate outcomes |
| Source: Purposeful Program Theory: Funnell & Rogers, 2011 | Source: Annie E. Casey Foundation | Source: ActKnowledge |
|
|
|
Other Resources for the Development of a Theory of Change
You can access the resource described below by clicking on the link provided in the title. Please note where the resource is a book as the link will only work if the book is commercially available. The link will take you to the international distributor, Amazon, where you can look through sections of the book before you buy it. If you live in South Africa, it is worth ordering it from local distributor Kalahari to save on import costs.| Resource | Description |
| Software: Theory of Change Online | Theory of Change Online or “TOCO” is an accessible, easy-to-use learning tool for creating and implementing a Theory of Change. It provides users with a flexible drawing canvas for building, editing, and soliciting feedback. |
| Book: Purposeful Programme Theory: Effective Use of Theories of Change and Logic Models (Sue C. Funnell & Patricia J. Rogers) | “Between good intentions and great results lies a programme theory—not just a list of tasks but a vision of what needs to happen and how. Now widely used in government and non-profit organisations, programme theory provides a coherent picture of how change occurs and how to improve performance. Purposeful Programme Theory shows how to develop, represent and use programme theory thoughtfully and strategically to suit your particular situation, drawing on the fifty-year history of programme theory and the authors’ experiences (Over twenty-five years)”. |
| Website: www.theoryofchange.org | A Theory of Change community with various resources for the development of a Theory of Change as a complete guide to programme design. |
| App: Evaluation Glossary | A handy reference of over 600 evaluation, programme planning and research terms and definitions. |
| Video: How to create a flow chart in Microsoft Word | Youtube video: Creating a Simple Flowchart in Microsoft Word, by Professor Floyd Jay Winters (uploaded in 2011). |
| Tools to assist with the preparation of a literature review | We provide links to a large number of resources that can assist you with the preparation of a literature review in the text of our Tried and Tested article: ‘Quick guide for doing a literature review for a situation analysis’. We don’t provide these links separately in the toolbox because many are specific to doing a literature review and not to the development of a Theory of Change per se. |
Step 2: Fill the Gaps - Attributes and Assumptions
If you have completed Plan: Step 1 you will have developed a Theory of Change in the form of an outcomes chain. An outcomes chain is typically a brief and visual summary for communication and presentation, but on its own is not sufficient for evaluating a programme. It is important to describe attributes (keyword definitions) because they unpack the meaning of outcomes. You also need to describe your assumptions or expectations about factors that affect the achievement of your outcomes. Identifying these assumptions could help you to avoid challenges that can undermine the success of your programme. It is also critical to consider when you want to evaluate the success of the programme.
What do We Mean by Attributes and Assumptions?
Attributes Attributes refers to the features of an outcome. It basically unpacks the meaning contained in the key terms included in the outcome. For example for the outcome: ‘Improve the sense of belonging of vulnerable children’, the key term, ‘a sense of belonging’ might be defined as: ‘the child have friends’; ‘the child attends school’; ‘the child participates in community sport/other recreational activities’; ‘the child has a close relationship to his/her caregiver’; etc. In the same way we could describe the attributes of the key term ‘vulnerable’: A programme might adopt a working definition where vulnerable is defined as being a maternal orphan. The key term ‘children’ might be defined as 0-14 year olds by a particular programme. Assumptions Programmes are based on certain expectations about factors that can affect the achievement of its outcomes. Some of these factors are generic and hardly need mention (for example, high quality management of the project). Some are unique to the programme and range from critical pre-conditions for success to circumstances that could limit the ability of the programme to achieve its outcomes. Some of them will be within the control of the programme (to a certain extent) and some not. Either way, these assumptions are important to consider when evaluating a programme. Examples: Outcome: Peer educators can effectively communicate information around the spread of HIV/AIDS.
How to Identify and Describe Attributes and Assumptions
Attributes
Source: Purposeful Program Theory: Effective Use of Theories of Change and Logic Models., Sue C. Funnell & Patricia J. Rogers; page 211
Assumptions
Each outcome needs to be reviewed to identify and describe the expectations of the programme relating to “helping” and “hindering” factors in successful achievement. Asking the following questions might be helpful.
- What is going to be critically important for the achievement of the outcome? What needs to be in place? What circumstances are important? What are critical activities or level of quality?
- What is going to threaten the achievement of this outcome?
- What are our assumptions/expectations about these factors?
- To what extent can we control it?
You can also do a SWOT analysis for each outcome to identify the factors affecting outcome achievement and the organisation’s level of control over it. A SWOT analysis is shown below.
We provide a SWOT analysis template as part of the toolbox for this step. You can download it here.
Possible sources of information that can inform the process of identifying assumptions are:
- Research, literature and other sources of timely information
- Experience of the programme or similar programmes
- Experience of the programme staff and other stakeholders
Documenting attributes and assumptions
It will be easiest to document the attributes and success factors in a table:
We provide a template for such a table in the toolbox for this section – download it here. Also read through the case study where we discuss the attributes and assumptions for the New Beginnings Development Centre. We also provide an example of a completed assumption and attributes table as part of the case study.
Tools
| Template: Document Attributes and Assumptions | |
| Template: SWOT Analysis | Template: SWOT Analysis |
PLAN - Step 3: Pin Down Your Plan of Action
Why Your Action Plan is so Important for Evaluation: Three Key Reasons
Knowing what the programme does is the most basic information required to understand what role the programme plays in producing the outcomes
The action plan is providing information on how the programme is supposed to work ‘in theory’. How it ‘actually’ works can then be compared to determine the extent to which the programme is implemented consistently with this theory. We call this testing programme fidelity.
Identifying the connections, or missing connections, between outcomes and activities (or missing actions) is important for effective programme implementation and to test the logic of the intervention. Can it work?
Important Things to Know About Representing Your Action Plan
Your action plan is critical to the achievement of your outcomes. This means your representation (documentation) of the plan will also provide a summary of your short and long term outcomes.
There are various ways to represent your action plan and the model that you choose, or create, is largely dependent on your preference. However, the complexity of your programme and external requirements from funders or other stakeholders might also influence your decision. The model most widely used, due to its longevity, is the logic model or log frame.
If you google ‘logic framework’ or ‘log frame’ you will find a large number of resources and log frame development guides. Although there are similarities, many of the methodologies and subsequent frameworks can be be very different which is confusing. Ultimately, a log frame is simply a way to summarise the most important aspects of your programme plan and logic in a way that makes sense to you and others. This has been interpreted in different ways by various organisations and individuals which is why there is such diversity. For many organisations, it is a giant step forward to have programmes documented in a simple way – this is why we suggest that you stick to what is straightforward and don’t get lost in the technical details/ arguments around log frame approach.
The ‘traditional’ option
Below is an adjusted log frame template suggested by the W.K. Kellogg Foundation who have taken a fairly simple approach. Although it does not contain all the detail of your Theory of Change (such as the assumptions and attributes) and cannot display the complexity of the causal relationships between your different level outcomes, it provides a concise summary of your resources/inputs, activities, outputs, outcomes and impact.
Have a look at this infographic for a simple, visual explanation of the terms used in the log frame.
A good alternative
As mentioned, in the absence of a Theory of Change or outcomes chain, the log frame is often criticised because it follows a linear and pipeline model of progression from resources to impact. It cannot express the often complex causal relationships between outcomes. This is not necessarily a problem when you are simply summarising the most important elements of your programme and do have an outcomes chain available. However, if you also want your model to illustrate how your programme works, you could elaborate on your outcomes chain and simply add boxes and arrows for resources, activities and outputs as shown below.
Read through the case study for this section and have a look at the examples (we provide both a table and a diagram) for a pratical illustration of how to represent/document your action plan.
The Four Steps to Identifying and Documenting Your Action Plan
- Remind yourself of the relevant definitions (resources, actions and outputs) and check the infographic if you need help.
- Use your summary of outcomes with attributes and assumptions (See Plan – Step 2) as a checklist to ensure that you consider all the factors involved in the achievement of each outcome.
- Identify activities to achieve immediate outcomes. In most cases, your immediate outcomes will lead to the achievement of intermediate outcomes and intermediate outcomes will eventually lead to impact. For each activity (or cluster of activities) identify the outputs that you will be producing and the resources that you will need for completion.
- Document this in the programme representation model that works for you. See Important Things to Know in Representing Your Action Plan for more information.
Remember to have a look at some of the resources provided in the toolbox to assist you with the representaton of your action plan. We also provide a template for a logframe table.
Further Resources on Log Frames and Tools to Help You Represent Programme Theory
| Resource | Description |
| Template: Logframe Table | Template: Logframe Table |
| Book: Purposeful Program Theory: Effective Use of Theories of Change and Logic Models (Sue C. Funnell & Patricia J. Rogers) | See pages 229-292 for a discussion on developing a theory of action and representing your programme theory. |
| Guideline: W.K. Kellogg Foundation Logic Model Development Guide | The W.K. Kellogg Foundation Logic Model Development Guide provides an orientation to the underlying principles and language of the program logic model. |
| Video presentation: Usable Knowledge online logic model training | Learn what a logic model is and how to build one through a narrated presentation by Usable Knowledge. |
| Video: How to create a flow chart in Microsoft Word | Youtube video: Creating a Simple Flowchart in Microsoft Word, by Professor Floyd Jay Winters uploaded in 2011 |
| Software: Lucid Chart | This is an intuitive and collaborative diagramming, web-based software programme. It does not allow for automatic layered documents, but you can always create several pages and insert hyperlinks. You can work on the same chart at the same time with people in different locations. There is an option of taking out a fourteen day free trial; from this it costs around R50 per team member per month and so is fairly affordable. Additionally, you will probably not need it every month. |
| Software: Creatly | Offers similar features to Lucid Chart, but provides thousands of ready-made templates for users to work directly from. There are very good pricing options (including one version that is free subject to some limitations). One of these limits is that all diagrams have to be public and only three people are able to collaborate at one time. Another great option is Creatly Desktop which allows you to work offline and automatically sync work to the cloud allowing everybody to collaborate. |
Step 4: Determine What Can/Should Be Evaluated
There is good reason to spend time mulling over and working on your programme theory; this leads to a markedly improved programme design and is a useful guide for the programme management. Above all, you simply cannot evaluate a programme without understanding what it is supposed to do and achieve. If you have completed the previous three steps, or have a well developed Theory of Change and action plan/outcomes chain/log frame, you will be able to identify evaluation questions to guide the development of your Monitoring and Evaluation (M&E) plan.
Four Good Reasons to Identify Initial Evaluation Questions
Identifying initial evaluation questions (which might change or expand during the process of programme implementation) is very helpful for the following reasons:
- It will assist you to identify ways to measure the progress that your programme is making. If you have never identified indicators before, you will find that identifiying evaluation questions first is very helpful as a step to thinking about indicators.
- You won’t measure everything all the time; these questions will simply assist you to think about what needs to be measured when. At the end of your programme you might find that you needed to collect information systematically over time to be sufficiently able to understand and explain the effectiveness of your programme. It is logical and more appropriate to measure some outcomes after a period of implementation has passed.
- Evaluation questions are the first step in keeping your data collection activity-focused and as simple as possible, ensuring high quality work.
- Measuring requires time, money and expertise; knowing in advance what is going to be required helps you to plan and budget for such activities.
What Should You Evaluate?
There are various definitions for evaluation, but the core meaning is to assess something in a systematic way in order to understand its value. This implies learning about its strengths and weaknesses which is information that can be used in various ways to adjust or make decisions about the programme. At a practical level evaluation comes down to asking simple but critical questions to enable us to learn and become accountable.
Accountability is often associated with accountability to funders. While there certainly is some accountability to funders, of most importance is the ethical responsibility organisations have towards their beneficiaries, to ensure that programmes really are meaningful. In most cases ‘beneficiaries’ are not just beneficiaries of programmes, but participants and stakeholders who have made an investment in the programme. This might be through labour, active support and participation or simply that they invest hope in improved living situations or life chances.
There are many questions that can be asked and aspects of a programme that can be evaluated to gather useful information and insight. However, in terms of ethical accountability, it is important that organisations ask and seek answers to the following (evaluative) questions:
- Is the programme effective?
- How effective is it?
- Why is it working or not working? (or why did it work or not work?)
- Were there unintended side effects?
- Who benefited most?
- Who was harmed?
In order to answer these questions an evaluation will need to look at:
a) The impact of the project
This refers to the longer term effects of the programme beyond immediate results. This is likely to be a comparison of a group of people who participated in, or were beneficiaries of the programme, with a group of people who were not involved. In order to understand why the programme is not having an impact, the evaluation will also need to assess the achievement of each of the outcomes in the outcomes chain.
b) The programme implementation processes (monitoring)
This is an assessment of whether the programme is being implemented as it was designed to be implemented. This is an ongoing monitoring system for the programme and helps determine whether or not the milestones and deliverables are on schedule.
c) The programme theory
This assesses whether the programme theory is sound and the proposed actions or approach are able to lead to the stated outcomes. This will also contribute towards an understanding of why a programme is, or is not, effective.
How to Identify Initial Evaluation Questions
It will now become clear why it was worth the time spent developing a Theory of Change and an action plan; identifying evaluation questions now is easy. You simply refer back to your log frame to help you determine appropriate questions for your evaluation. In terms of process you will ask whether or not inputs have been sufficient and timely, activities took place and outputs were successfully produced. In terms of impact, you will ask whether your outcomes have been achieved and ultimately whether the programme has made a lasting difference in the lives of the beneficiaries. In order to understand why the programme or intervention might not be performing as expected, you can ask questions about the validity of the theory of action and change and the underlying assumptions of the programme.
Useful resources
| Template: Indicator, Target and Means of Verification Table | Template: Indicator, Target and Means of Verification Table |
Step 5: Find Indicators and Set Targets
You have arrived at the challenge of measuring! How will you be able to answer your evaluation questions? How will you know that you are making progress in achieving the change you want to bring about? If you have, to what extent?
Indicators: Basic Concepts
What are indicators? | We cannot directly observe social phenomena and yet know social phenomena exists and is present or absent under certain circumstances. How do we know this? There are indicators. Let’s look at the concept of social cohesion as an example. We know that social cohesion is present or absent by looking at what people report about trusting others in their community; how satisfied they say they are with life (supported by the suicide rate); the presence or absence of pro-social behaviour (such as volunteering, helping strangers or donating money); as well as voting rates, as high voting rates could indicate that people care and want to participate in the management of their community/country. |
Attributes/Characteristics of Indicators
They are imperfect | Perhaps you can point out more indicators of social cohesion or you would like to challenge the ones that have been listed above. This is an important aspect of indicators – they are approximations, imperfect and vary in validity and reliability[1]. Sometimes a single indicator is not sufficient to say that a certain change has occurred or that a phenomenon is present or absent. For example, a high voting rate on its own does not necessarily imply high social cohesion; it might be the opposite if people have been coerced to vote. In such cases you will need to include additional indicators and develop a set which together illustrate the existence (or absence) of the phenomenon. |
They can be shared | Indicators are sometimes obvious. For example, an indicator for reduced school drop-out rates would be the school graduation rate. Other times indicators can be complex and require a lot of research to identify/develop the indicator/indicator sets. It is in our interest to develop standard indicators of social phenomena for people to measure. This allows us to compare information across different programmes, geographical areas or implementation circumstances. You will see that in many fields, such as health or economics, standard indicators have been developed and it is only a question of selecting an appropriate one. If you are doing something that is unique in nature, you will need to develop your own indicators. See the Tried and Tested section of this step for more information. You can look for indicators shared by organisations in the indicator library available under the tool section of this step. If appropriate, you can use the indicator as is or adjust it and use it as a guide for developing your own. Please be sure to contribute your indicators to this communal library. |
The purpose of indicators is to measure something | We look to indicators to answer the question ‘is it there’ or ‘did it happen’? Thus, the point of using indicators in evaluation is to help us observe and measure change. Did something happen that did not happen before; to what extent did it happen (was the change big or small); how often is it happening (is the change sustained or not) etc.? To answer these questions implies the counting and comparison of quantities at different points in time. Therefore indicators are generally expressed in terms of numbers or percentages (number of…; percent of…; ratio of…; incidence of…; proportion of…). Researchers would say it is quantitative. Sometimes people refer to qualitative indicators when they are measuring phenomena which they would define as ‘qualitative’ (such as being happy) and to quantitative indicators when they are referring to something they can obviously count (like the number of brochures that they have distributed). The reason why indicators are generally expressed in a quantitative terms is not related to the nature of the phenomena that you are investigating however, it is because the type of questions that we are trying to answer through indicators requires counting (it happened yes (1) or no (0); it happened to a large extend (90%) or it barely happened (3%). That does not mean that qualitative information is not useful to evaluation, it answers different questions though and therefore it is not expressed as indicators (read here about the usefulness of qualitative investigation in evaluation). It does mean that when you are selecting indicators you need to be sure that there are ways of measuring (counting) the occurrence of the indicator – for example, levels of happiness are generally established through life satisfaction surveys. |
Indicators are complemented by: – targets | Targets (to establish the level of change needed to qualify success) How much change will be enough to achieve the impact that you want your programme to achieve? If you offer a skills training course that capacitates people to find employment, is it sufficient if only two out of ten find employment? Or would you say your programme has achieved this outcome when at least seven or eight out of ten people find employment? Because change and social phenomena can occur in ‘quantities’ (ranging between a lot and a little), it is important that you qualify at which level of change you would say that your programme has successfully achieved (the outcome(s)). Because you are using an indicator to establish (measure) whether the change has happened or if certain social phenomena are occurring, your targets need to relate to your indicators. Baseline (to establish if change has happened and to inform targets) If you want to establish whether something has changed, you need to compare it to how it was before your intervention. To do so, you need to know the initial status before your intervention – we call this a baseline. Basically, a baseline requires you to do all the measurements, for selected indicators of outcomes that are key to showing that the intervention is achieving what it set out to do, before you start implementing the programme. You can keep those numbers on record for future comparison when it comes to evaluation and you can also use it to inform your targets. A comparison group (to establish if your programme is responsible for the change) It is fine to establish whether change has happened using indicators, targets and baselines, but how do you know that this change happened as a result of your intervention? In some cases, it is obvious and there might be no need to investigate further. This might be seen in a programme that establishes a children’s hospital in a needy area and as a result more children are receiving medical care when they need it. However, interventions with less concrete outcomes makes it more difficult to report that the programme is responsible for or has contributed to the change. For example, a programme designed to improve a love of reading might measure the number of books that children say they read in a month (an imperfect indicator, but we will use it for the sake of an easy example). If the programme notices an increase in the number of books read compared to the baseline, it could be as a result of their programme. However, if the SABC had started broadcasting a programme promoting reading, or the Department of Education improved school libraries, we cannot really be sure that the ultimate change was solely due to the example programme given. The best way to establish attribution would be to look for the occurrence of your indicators in two groups: the target group that you have been working with and a group that you know for sure has had no exposure to the programme. Researchers would call the second group a ‘control group’. This process of comparison is called quasi-experimental or true experimental research. |
Important Concepts to Understand About Measurement
Measurement entails ‘how’ you will establish the performance of your indicators. In other words, the extent to which your programme has achieved its outcomes. It answers questions around the impact of your intervention and whether or not your programme has caused or contributed to change, if it has been achieved.
Measurement varies in difficulty. Some indicators relate to your outputs, the products and services produced through your programme representing evidence of programme activities. These will mostly involve systematic counting of products and services delivered and ideally you should develop tools and data collection methodologies yourself. They are related to the specific operations and management of your programme/organisation. For outcomes with indicators that are more complex to measure, it might be best to consult with a researcher that has experience using different types of measures and research methodologies. As a start, you could post your question on this website as we have a network of evaluators that are supporting this community.
Remember that measurement is empirical – this means it is knowledge that is derived from observation or experimentation and is not based on theory or logic or individual thinking. Keep in mind that there is a difference between people’s perceptions and information based on facts which is objectively verifiable information. For example, you might come up with different answers when you ask people if they think they have learned something and then test them on that knowledge or new skillset.
For evaluation of social programmes you will generally try to measure (please watch the video for some practical examples):
| Indicators that are linked to an outcome that talks about ESTABLISHING OR DOING SOMETHING NEW. As there is no comparison involved, you should be able to measure this in a single instance. | Non experimental researchCan illustrate that something took place and describe it. This includes some cases where its occurrence is associated with the occurrence of something else, like your programme activities. Although it might allow you to say that two or more things happened concurrently, it cannot say that one caused the other one. If you are making that conclusion you are guilty of some logical inference at best and wishful thinking at worst! |
| Indicators that are linked to an outcome that talks about ACHIEVING A MINIMUM STANDARD. As you are not comparing over time, but against the standard, you should be able to measure this in a single instance. | |
| Indicators that are linked to an outcome that talks about INCREASE/IMPROVE/DECLINE OVER TIME. If you speak about an increase/improvement you will need to compare your indicator against two points in time. This means that your measurement will entail some kind of pre- and post-testing, or knowledge about the baseline conditions. | Quasi-experimental research:Pre- and post-testing and comparison to the baseline involves comparing the participants against themselves before and after the programme. If you are testing for objective factors and not only for the perceptions of people (for example, you are actually testing the knowledge of participants and not only asking them whether they think they have learned something), you should be able to effectively say if change has occurred or not. If you can be 100% sure that the programme was the only factor influencing the changes that you measure over time, this could mean that your programme is the cause of the change. |
| Indicators that are linked to an outcome that talks about PROGRAMME EFFICIENCY AND ATTRIBUTION. You measure the impact or efficiency of your programme by establishing what the status of things would have been if your programme had never been implemented. Researchers call this the ‘counterfactual’. However, ‘how things would have been’ is clearly impossible to measure directly; it can only be inferred. One way of inferring this is by comparing the outcomes of those who participated in the program against those who did not participate. | Experimental and/or quasi-experimental research:Apart from pre-and-post testing, there is a variety of quasi-experimental research design that could be used to compare groups of people. The difference between quasi-experimental research and experimental research is that in the former, groups are pre-assigned based on some characteristic or quality. For example, in most cases of post hoc (after the fact) evaluation the ‘research’ group will inevitably be the group that has participated in the programme with their characteristics targeted by the programme. The challenge will be to find a ‘control’ group of people who did not participate, but would closely resemble the participants if they had not received the programme. With true experimental research the assignment of groups are completely random, a bit like the lottery winner selection if everyone only had one ticket. The programme is then implemented only in one group and comparison of the two groups is done when appropriate. Although randomised evaluations are the ‘gold standard’ in terms of evaluation, we know that many organisations don’t start with impact evaluation in mind, but rather with a passion to assist people who need help. Quasi-experimental designs are dependent on more assumptions than true experimental design. For example, that non-participants are identical to participants and are equally likely to enter the programme before it starts. If these assumptions hold true, this type of research gives us the correct answers to our evaluation questions. There are ways to do quasi-experimental research that will reduce the risks of biased answers. Have a look at this useful chart by JPAL which explains the different designs and the assumptions that underlie them. |
Different Types of Indicators
If you have completed Plan: Step 4 you will be selecting and developing indicators for most of your evaluation questions. These are already linked to different programme elements like outputs, outcomes and impact. We can label indicators relevant to outputs as ‘output Indicators’ and those relevant to outcomes, ‘outcome Indicators’. It is useful to know that the information we collect on our processes and outputs influences the routine monitoring of programme implementation. Information also influences outcomes and impact for the ongoing assessment of the effectiveness of your programme (the evaluation).
Top Three Criteria for Good Indicators
If you have not done so already, please have a look at the Knowledge section: Indicators: Basic Concepts, before you read further as this article builds on the information given there.
You will find many criteria for good indicators on the internet; some more comprehensible than others. We have studied them all and narrowed them down to the following three key criteria for good indicators. After reading this you can test the strength of your indicators on the ‘Test the strength of your indicator tool’ which you can access in the toolbox section for this step.
Criteria | Meaning | Example |
Valid | An indicator is valid when it accurately indicates the phenomena that you are trying to measure. It should be sensitive enough to pick up changes in the phenomena over time. | Life satisfaction has been found to be a valid indicator of happiness. Although high self-esteem is often associated with happy people, it has been found that high self-esteem in itself is not a valid indicator of happiness. The reason for this is that people who esteem themselves positively can experience periods of unhappiness. |
Reliable | The indicator will accurately indicate the phenomena if measured by different people in different places over time. It should be written clearly, using language that will be interpreted in the same way by anybody collecting information in any circumstance. The indicator is more likely to be reliable if it is very specific about what is to be measured. | An organisation was collecting information about schools participating in their face-to-face interaction programme. Their indicator was ‘number of schools registered”. (It is worth noting this could have been more specific, such as number of schools registered for x). Over time, besides the regular face-to-face programme, they also started to offer some events-based activities in which schools could participate. Some of their staff, who wanted to increase their performance on targets, decided to apply a broad interpretation of the indicator and so registered schools for participation in these ad hoc events. After a while, it was discovered that the indicator was no longer a reliable tool to assess school participation in the organisation’s regular face-to-face programme. The indicator was re-written, but as a result an unknown chunk of the school participation data became invalid as it could not be used as comparative data any more. |
Simple and Affordable | Every indicator involves an exercise in data collection that will involve your staff or programme participants. Data collection for the indicator should not be too burdensome on people and the amount of resources (funds, personnel, time) required should be reasonable. The more burdensome data collection is, the more likely it is that the quality of the data collected will be sub-standard. This is also a risk if the data collection process is too complicated. You need to be realistic about the time that is involved in collecting and submitting the data on the indicator. Incomplete data that is not submitted in time will affect your own and others’ understanding of your progress. Be aware that providing incentives to motivate people to collect data might lead to biased or false reporting. | Community clinics under the Department of Health (DoH) collect service utilisation data for certain predetermined age categories (such as 0-1 years; 2-6 years). A programme focused on the reproductive health of youth of a particular age category, which happened to fall across two of the DoH categories, wanted the clinic staff to recalculate the utilistation statistics each month in order to provide information for their indicator – ‘utilisation of reproductive health services’ for the age group relevant to them. This meant the clinic staff had to adjust their basic data collection technique and keep an additional template to capture and calculate this data. This was an increased workload and became a burden. This process was not institutionalised and so clinic staff would often forget to do it or do it incorrectly, often when staff rotated. In the end, the data collection required to measure this indicator was not realistic and had disappointing results. Additionally, clinic staff and evenutally the DoH found the programme irritating. |
Using Qualitative Information in Evaluation
“Not everything that counts can be counted” Albert Einstein once pointed out. Let’s take the example seen in the love and care that an African grandmother (often called a ‘gogo’ in South Africa) shows to a young orphan and try to imagine what the impact of this love will be on the life of this child.
If we say that a large aspect of M&E is measuring whether something has happened, and if measuring implies counting and comparing quantities at different points in time, what role does that which we cannot count play in evaluation?
In the example above, loveLife tried to establish the level of vulnerability of the orphan before a gogo would start interacting with him or her. This was done using a few practical indicators. Is the child attending school? Does the gogo suspect abuse? Is his family accessing a child support grant for the child’s welfare? loveLife then monitored the practical things that the gogo did to assist the child with the factors mentioned. For example, they assessed whether she would speak to the school headmaster and motivate for the child to return to school and whether or not she explained the requirements needed to claim the support grant for the family. After a few years loveLife would check if the original vulnerability score for the child had changed, i.e. was the child less or more vulnerable when compared to the period before the gogo was involved?
The above worked fairly well to establish whether the programme was doing what it was meant to be doing effectively. However, the most important impact of the programme (that of the love experienced by the child) cannot be counted or illustrated in quantitative terms. We know from research on early childhood development that early love creates a legacy of resilience in children which benefits them for a lifetime. The field of epigenetics illustrates that early love and caring actually changes the way that our DNA expresses itself. This has been dramatically illustrated through studying the effects of early deprivation on brain development.
By studying the relationship between the gogo and the orphan in a qualitative way, we can begin to understand what that relationship looks like and what it means to the gogo and the orphan. Over time we can gather a great variety of information on the child, ranging from school and clinic report cards and interviews with teachers and family members. We might even observe and interview the child at certain points in time. As we study this information in depth, the meaning of the early care and love might start to reveal itself in subtle ways. When the child is older he or she might be able to reflect and share with us how the experience of that love and care played an influence and what he or she thinks it may have meant to her life trajectory.
It is not viable do such an in-depth study of the lives of many participants. However, when qualitative research is done well, we are able to draw valid conclusions when we are evaluating programmes. The type of questions being answered are different and allow for a deeper understanding of the situation with greater insight into why and how something is the way it is.
Information and Data Collection Methodologies
There are various ways to collect information about your programme. We have provided you with an abridged version below.
Collecting Data On: | Methodology or Tool | Description | |
Activities/Output (the services and products offered by your organisation) | Monitoring system | A monitoring system generally takes a variety of forms, using procedures to document and count the occurrence of outputs and activities. Some of the tools that you might use are well known. For example, an ‘attendance register’ would be used to count the number of people participating in a training. Other forms and tools will have to be created by you; perhaps your system uses an online database with electronic forms that your staff complete as they deliver a service. You can look at our monitoring section for guidance on developing a monitoring system for your programme. | |
Outcomes and Impact (to measure the achievement of your outcomes and effectiveness of your programme) | These are also column headings (left = Quantitative methods, right = Qualitative methods): | A quantitative approach (involves asking information in a way that requires counting[1] and will give you information that is measurable and comparable) | A qualitative approach (provides you with information that will allow you to develop an in-depth understanding of the nature and characteristics of the phenomena that you are studying) |
| Interviews with individuals | Structured interviews allow you to repeatedly collect the same information from different people on specific, predetermined questions or concepts (variables). | In-depth interviews ask open-ended questions about a general theme, which allows various sub-themes to emerge through conversation. This allows you to develop a deep understanding of the theme or the participant’s perceptions of the theme. | |
| Group discussions | Focus group discussions are types of discussions led by a focus group moderator. They last approximately 60–90 minutes during which a group of 8–12 respondents are asked to comment on specific, predetermined questions or concepts using a structured moderators guide (questionnaire). | Group discussions are informal, largely unstructured, in-depth group interviews where multiple respondents give input. | |
| Observation | Observers record what they see and hear at the project site using an observation checklist. The observation may be of physical surroundings, ongoing activities, processes and discussions. | This is clasically a qualitative method, also known as field research/ethnography. It involves the detailed observation of people in a specific setting/location (often over an extended period of time) to obtain an insider’s point of view. This is done in order to develop a deep understanding of the theme being studied. | |
| Case studies | Not done for quantitative research. | A large amount/variety of general information (such as information gathered from in-depth interviews/group discussions/observation) is compiled on a limted number of cases and studied in-depth. This analysis shows emerging patterns and themes that allow a greater understanding of the subject being researched. Important note: An anecdote/letter from a beneficiary is not a case study – it might be one piece of information in a case study. | |
| Surveys | A survey normally involves a questionnaire or test which is designed to collect information, that is easily countable, on very specific questions/themes (variables). These questionnaires can also be completed using interviews, often seen in the proccess of collecting data for a census. The real issue in surveys is the fact that a smaller (sample) group of people are selected to be studied from a larger group and researchers hope to make generalisations about the entire group based on the information taken from the sample[2]. One way of doing this is to randomly select a large enough sample. Random selection means that a participant’s chances of being chosen is a bit like like winning a lottery. Every person has one ticket and an equal chance in selection. If you select enough people in this way you will find that the attributes of the chosen individuals are representative of the entire group from which they were originally chosen. Although there will be some limitations in terms of the strength of your study, you can also do informal small-scale sample (30-50 individuals) surveys where participants are not randomly selected. There are various methods of selection, such as purposeful selection of people in different categories. This is done on the basis of easy accessibility and appropriate statistical tests. | Not done for qualitative research | |
[1]This asks specific questions that do not require further elaboration such as “do you have a television – yes/no”. This often provides pre-determined options that participants need to choose between, such as “best thing to do with a ball: a) kick it; b) throw it; c) hit it” or it asks participants to rate something on a scale (e.g. 1 – 5, with 1 representing terrible and 5 meaning wonderful).
[2] Except when they do the census; in this case they actually try to count everybody (researchers would say a census represents a complete enumeration, or listing, of all units in a population)
How to Select, Identify and Write Up Indicators
Before moving on to the selection and identification of indicators, please work through the two knowledge sections for this step:
Identifying indicators is the last part of your M&E plan and has three components:
- Selecting/identifying and phrasing an indicator or set of indicators for each of your outcomes
- Setting a target for the indicator
- An initial identification of the source of the information for the indicator and the tool/method that you will use to collect the information
Selecting/or identifying indicators takes some time and thought and might involve some consultation with stakeholders. Your indicator list will be the blueprint that directs all of your data collection activities. Proving that change has happened often involves making comparisons over time and the effectiveness of your evaluation will decline if you change your indicators during the implementation process. It is worthwhile spending some time early on to get this right. If you have been implementing your programme for some time, but you are only preparing an M&E plan now, don’t worry. It is a bit like quitting smoking; the sooner the better and beneficial from the moment you do it.
Identifying Indicators
You may start to newly identify and write indicators if there are no standard indicators available, or you will simply look for and select standard, appropriate indicators (for example, those that exist for health and nutrition). Whether you are newly identifying or selecting from existing indicators, answering the following questions about your outcome statements/outputs or activities will be helpful.
- What would answer my evaluation question? (Remember Plan: Step 4, where we identified evaluation questions)
- How will I know that the outcome/output has been achieved or that we are making progress in its achievement? Or you might ask:
- What will tell me, or point to the fact, that the outcome/output has been achieved or that we are making progress in its achievement? Or you might ask:
- What will it look like when x/y has happened? Another way of thinking about this might be:
- If I were a visitor, what would I see, hear or read that would tell me this phenomenon exists or has happened?
If possible, you should identify various indicator options for each outcome and then choose the one (or compile a set of indicators) that is strongest. Remember that indicators are approximations and they vary in validity and reliability. Keep the criteria for ‘good’ indicators in mind when you select them.
How should you phrase an indicator?
The questions above probably helped you to write up indicators with ‘direction’. For example, you might have said that the indicator for an outcome of ‘improved education for South Africans’ might be ‘an increase in the national matric pass rate’. For an outcome of ‘safer sexual behaviour’ you might have said the indicator should be ‘a decline in the rate of sexually transmitted diseases (STDs)’.
This was an exercise in identifying the indicator, however, your indicator should be able to show you change in any direction, not only the direction that you would like it to. When you phrase your indicator, you will not usually include the desired direction of your indicator (which is already captured in your outcome). In the first example above, we will simply state the indicator as ‘the national matric pass rate’ and in the second example, ‘the STD rate’. However, you might find it more intuitive to write your indicator and include direction; this is really not a big issue, as long as you realise that measurement should be able to show change in either direction.
Generally your indicator will include:
- The unit being measured. For example, ‘Number of…; Percentage of…, Ratio of…, Incidence of…, Proportion of… Rate of’. The unit can either be described in terms of the number of people involved or how many times we are observing certain phenomena happening. Note that we sometimes include both the percentage and the number in the phrasing of an indicator, because a number in itself does not indicate the magnitude or rate of the result. An example is 5 of 10 or 5 of 200; the percentage itself does not indicate the size of the result. One could ask, “30% of what?” by way of illustration.
- The subject/phenomena being measured. This is the essence of the indicator and states exactly what is being measured. Examples are endless and directly relate to your intervention. This might be knowledge or skill, access to services, presence/absence of a specific disease or health promoting behaviour and so forth.
The way you phrase an indicator can impact its reliability so be sure to keep the criteria for good indicators in mind when you are phrasing them. After you have identified and phrased an indicator you can test its strength on our Indicator Strength Tester Tool (see the toolbox section).
Here are some examples:
Next, you will need to link targets and means of verification to your indicators which will be explained in the second ‘Tried and Tested’ article for this step (Plan – Step 5).
How to Link Targets and Measures to Your Indicators
The last part of your evaluation plan is to link targets to your indicators and to specify the ‘means of verification’ for each indicator. If you have not done so already, we strongly advise you to go through the ‘Knowledge sections for this step (Plan – Step 5): Indicators – Basic Concepts’ before continuing and ‘Important Things to Understand About Measurement’.
Targets
The target is your estimation on ‘how much’ of your outcome need to be achieved within a certain period of time to achieve the intended impact . You can also set interim and long term targets to realistically reflect achievable performance goals for your programme. Because the means to measure the achievement of your outcome is your indicator, the target will relate to your indicator. If your outcome speaks of improving or increasing something, you might establish a baseline to allow for comparison and inform your targets. Targets should also be influenced by research and the experience of similar programmes, to ensure they are realistic.
Means of verification/measures
The means of verification/measure is a short reference (not a complete methodology) showing ‘how’ you will investigate and establish the performance of the indicator. Your measure could reflect the tool you would use if you know of a relevant tool that exists. For example, to identify pregnant women with mental health risks. you might use the ‘Edinburgh Postnatal Depression Scale’. Alternatively a measure can easily be created in some cases. For example, to measure attendance of training an ‘attendance register’ is used. If there are no relevant standard tools you might indicate your methodology for collecting the information, such as ‘face-to-face interviews’ or ‘pre- & post-tests’. Sometimes there are standard available statistics that can be the source of your data for the indicator. If, for example, your indicator is ‘the HIV prevalence rate for a certain population group’ your means of verification might be the ‘National Antenatal Sentinel HIV & Syphilis Prevalence Survey for x year’.
It is important that your outcomes, indicators, targets and means of verification align. For example, we sometimes talk about an increase in performance. This implies comparison over time, but the target does not reflect comparison. For example:
- Outcome: Increase in learner literacy level
- Indicator: Learner performance on literacy test
- Target: 80% of learners must pass (Implying a once-off measurement – it should have been a x% increase from baseline for example)
- Means of verification: End of year test
As a general guide:
- When you speak about an increase/improvement/decline in your target should say by how much it should increase and from what status it should increase over a given time period.
- When you speak about something new happening, or a certain standard that should be achieved, your target will specify the minimum standard that you are looking to achieve. It can either be related to the number of people or to the number of times that they should do something – or both.
Examples:
You can complete your evaluation plan by adding your targets and means of verification to your indicators and outcomes table. If you have not completed it yet, download the template here. Please read through the case studies for this section (Plan-Step 5) and have a look at the example for the New Beginnings Development Centre before you start assigning indicators, targets and measures.
Your complete Monitoring and Evaluation plan should include:
- Your Theory of Change/outcomes chain diagram and/or;
- You attributes and assumptions table
- Your action plan, which might be represented in a log frame, as a table or an expansion of your outcomes chain
- Your indicator, target and means of verification table
Useful resources
| Template: Indicator, Target and Means of Verification Table | Template: Indicator, Target and Means of Verification Table |
Guides to help your NGO thrive
Navigating the technicalities of establishing, registering and running an NGO can be intimidating. We provide free guides to help your organisation grow and thrive. With easy-to-follow instructions on important topics from how to establish an NGO to governance, monitoring and evaluation, as well as useful tools and templates, we’re here to help you!
Are you looking to establish a non-governmental organisation (NGO) or searching for resources to strengthen your pre-existing organisation? We’ve curated a guide that will help you think through the legal, financial and technical questions associated with setting up and governing an NGO in South Africa.
Monitoring and evaluation (M&E) is essential for NGOs to determine how effective the work they’re doing is and identify areas of improvement. We’ve put together an easy-to-follow guide for all levels of experience including tips, tools and advice from experts.
Many NGOs may feel intimidated by the Protection of Personal Information Act (POPI Act or POPIA). To assist you, we have compiled a guideline that unpacks, in simple language, the most essential aspects of the Act that NGOs need to know and understand in order to take the first steps towards becoming compliant.
