Monitoring & Evaluation
Measuring for success, impact and learning
Success or failure in sustainability projects is also determined by reliable and accurate data. That is why Hanns R. Neumann Stiftung (HRNS) is putting high efforts into its Monitoring & Evaluation (M&E) system. It guides the overall development of our projects and programs and is key to the relationship with our partners. Measuring success is the basis for achieving this.
How important is M&E for HRNS?
Monitoring and Evaluation is of crucial importance for HRNS because it helps us improve our programming. It also serves as an accountability mechanism to our donors and supporters as the data collected serves as proof of what we have achieved with the funding that donors and supporters have entrusted to us for the beneficiaries.
Via our monitoring, we track all our trainings for farmers and youth but also check the adoption of the training contents and then assess, whether this affects yield, income, or food security of farming families. If it does not, we are able to identify where in the “impact chain” the effects stopped materializing and can then investigate the causes together with our colleagues, farmers, and other stakeholders in the region.
External evaluations help us to look at our projects and programming from a bird’s eye perspective. Via evaluations we not only have the functioning of our impact chain checked in detail by independent third-party experts, but also verify if the problem we chose to address with the project was indeed relevant for the country context, if the problem-solving approach chosen was the most effective and efficient, and if our projects have the prospects of leading to positive, long-lasting, and sustainable impacts in the lives of the beneficiaries.
Thus, via monitoring and evaluation we find answers to the following questions: What went well, what did not and why? What shall we continue doing and what can we do better in the future?
The crucial importance of M&E at HRNS is reflected in the organizational structure of our offices: Our M&E personnel all have a direct reporting line to the Country Managers or the General Manager in the case of our headquarter. This way, results and learnings from M&E could directly be shared with the management and shifts in strategy or programming could immediately be applied by the management if needed.
How does HRNS collect data?
We collect data almost every time we are in contact with the smallholder families we work with: At the time of registration, we register the size of their family, land and coffee plots, the age of the head of household, the date of registration and from which specific projects they will benefit.
We track the number, type, location, and attendance rate of our trainings and the specific subjects taught in that day. We collect this data for each farmer we work with and every training we conduct, to be able to prove, how our funds were spent in each country and each project.
We then also visit a representative sample of farmers to assess how many adopted Good Agricultural Practices (GAP) that were taught in the trainings they attended. This assessment is done by both, asking the farmer questions via a structured survey and by observing if the practices due in the month of surveying were implemented. When conducting these surveys, we usually visit between 100 and 400 farmers, depending on the total number of beneficiaries and the funding available.
We also ask, if they observed any changes in their yield, household income and, if they are part of projects that address gender relations, food security or health, we also ask for changes in these domains.
As most of the smallholder farmers we support are organized in farmer organizations, these play an important role in our programming and thus in our M&E. On the one hand, we are interested in the composition of their members regarding gender and age in general and in leadership positions. On the other hand, they bookkeep how much coffee of their members was marketed and sold. This helps us to assess their effectiveness and to check the validity of data the farmers shared with us.
We also support the Farmer Organizations in assessing and improving their organizational capacity and here, Monitoring and Programming are directly interlinked: Once the capacity level of the farmer organization is assessed for our monitoring records, we subsequently discuss which measures they could take to further improve their services for their members, and thus their ranking in the next assessment, which is usually after one year.
What sort of data does HRNS collect and how are they shared with donors and the public?
Data is becoming a driver in the industry as roasters and donors want to be increasingly informed via aggregated quantitative data and partly also involved in data interpretation. Thus, we collect quantitative data on trainings, adoption of practices, effects on the livelihood situation of farming households like yield, income, and food security. We also collect quantitative data on the effects of climate change on the plots as well as how resilient their farms are towards climate change. In addition to the quantitative data, our colleagues in the country offices do more in-depth qualitative interviews with specific farmers who are willing to share their story and experiences more in detail with the public.
Data on trainings and beneficiaries per region and project is collected across the whole HRNS network on a quarterly basis. It is visualized by the headquarter in easily interpretable graphs and diagrams via PowerBI. The visualized data is shared with all colleagues within our foundation on a quarterly basis.
Data on the adoption of Good Agricultural Practices (GAP) and effects on the livelihood situation of farming households is usually collected annually or three times per project: at the beginning, in the middle and at the end. This data is being analyzed, visualized, and interpreted usually by the country M&E staff, partly with the support of the headquarter. It is always shared with the colleagues involved in the specific project, i.e., all field staff, country managers, program and partnership managers at the headquarter and of course the respective donors and supporters of the project.
Data relevant to all HRNS’s work and exemplary data from specific projects of interest is also shared with the public, either on our website or in specific publications presenting our work and impact.
Are there challenges in data collection and how does HRNS overcome them?
One challenge is the timing of the data collection, especially the surveys. Since these surveys can take up to an hour or more of the farmer’s time, we try to do them when the farmer has a break, i.e., not during harvest.
Another challenge is the reliance on memory of the farmers. Many of the farmers do not take accurate records of input costs, e. g. for (organic) fertilizer, seedlings, additional workers during harvest, nor do they bookkeep their exact yield, income, or net profits. Although keeping a farmer field book is one important component of many of our projects, not all farmers immediately apply the bookkeeping at a necessary level of accuracy. Consequently, a lot of questions on yield and income are answered based on the farmers’ memory. Thus, we try to do the surveys right after harvesting so that farmers still remember well how much yield they got and how much income they earned.
A third challenge is, that we work with very high numbers of farmers who often live very scattered across a vast rural area. This makes it difficult to visit all of them regularly and to do surveys. To overcome this challenge, we found the compromise of only surveying a representative sample of farmers. This representative sample is drawn randomly, so that each farmer has the same chances of being part of the survey. We usually visit this same sample at the beginning, in the middle and at the end of the project.
Data qualityis key
How does HRNS ensure quality of the data?
One of the most important determinants of data quality is that each farmer understands the question we ask in the same way and responds accordingly. Each country and partly also each region has different terms and words to describe the same agricultural practice or climate related phenomenon. When we collect data with farmers in a specific region, we make sure to use the wording that the farmer will understand. Otherwise, we would run the risk of measuring different things with different farmers. To guarantee a common understanding of our questions, our M&E colleagues in the countries make a pre-test before conducting surveys. This encompasses discussing with a hand full of farmers how they understand the survey questions and double-checking if their understanding fits with the understanding of our M&E colleagues. If questions or terms lead to diverging understandings, they will be adjusted.
Another important step towards good data quality is training of enumerators who will help our M&E colleagues to collect the data. During the trainings, they learn the survey protocol and that answers of farmers shall not be influenced in any way by the enumerator. To make sure our enumerators comply with the protocol, they are closely supervised by our M&E staff and spot checks are conducted.
To end up with robust and comparable data, we also need to be aware of different units of measurement. Each country has different units of measurement for yield, different currencies for income and the coffee is sold in different forms, either as fresh cherries, dried cherries, parchment, or green bean. When we ask for the yield, we make sure to also note the unit of weight the farmer was referring to as well as the type of coffee in which the yield was weighed. This is important to be able to compare the numbers with other farmers’ data and to correctly interpret the findings. When using digital data collection devices, we use inbuilt conversion calculators which would allow the enumerator to indicate the value and select the unit of measurement used by the farmer, e.g., acre. The value is then automatically converted to hectare if this were the unit of measurement we would like to work with in our analysis.
What happens to the data once collected?
Once the data is collected, it is cleaned. This means, all data is converted into the correct and uniform format and duplicate and invalid values and survey sets in which most questions have not been answered are deleted from the data sets. It is then processed by converting the data from individual respondents into aggregated data. This is usually done by summarizing values from different respondents and/or survey questions into one average value or score, total aggregated value, or percentage. These are often the values of our project indicators but can go beyond that.
The data is then visualized via graphs and diagrams to ease analysis and interpretation of the results. The findings are then shared by our M&E colleagues with the project teams and respective managers, and jointly discussed and interpreted. We ask them: Where are values developing as intended, where not? Do these results mirror what you have perceived in your work with the farmers and when you visited the fields? What are the causes that values do not develop as intended? How can we address these causes in our programming? What are the next steps to apply these measures?
The results together with our interpretation and, if needed, suggested measures to adjust programming are then shared with our field teams on the ground and our donors in the next report.
What is the role of digital tools, how do you use them and what are the future plans for them?
Digital tools play a critical role at HRNS since they help us to work more efficiently. We apply digital tools at several stages of the M&E cycle: We design a survey online and can access it once we stand on the plot with a farmer to enter the data in a tablet or smartphone. The programmed surveys have a functionality to validate data values when they are entered in the system. For example, we can predefine the range of values that would be valid and automatically alert the enumerator if values are outside the valid range. Unrealistic data is then detected during the survey and errors can be corrected immediately. Digital data collection also helps us save time that was previously needed to manually transfer answers from paper-based questionnaires into excel or other data base formats. Via digital data collection, enumerators can upload the collected data with one click once they have internet connection again. Our M&E colleagues can access, process, and analyze that data right away. Most of our country teams use the open-source mobile platform CommCare for survey design and digital data collection.
For processing the data at the headquarter we use the ETL (Extraction, Transformation, Loading) software Alteryx. Since we work with six different regional offices, the data sets we receive partly differ in format and type. Alteryx helps us to clean, format and merge all six into one joint HRNS wide data set that can be analyzed, visualized, and compared to data of previous years and reporting periods. The joint data set, unified by Alteryx, feeds into automated reporting templates in PowerBI and is visualized in diagrams and graphs accordingly so that we can see at first glance what the data wants to tell us. These four different steps, questionnaire design and data collection via CommCare, data cleaning and unified formatting via Alteryx and data visualization via PowerBI, are facilitated by digital tools and not only help us ensure higher data quality but also save us considerable time and manpower. What remains fully analog is the way the whole M&E process starts and the way it ends: a conversation. It starts with a talk with farmers, and it ends with a discussion with the program teams. We aim to keep it this way as the relationship with the smallholder families and our colleagues is the core of our work and key to success. In between we use state-of-the-art technology. For the future, we aim to link our own data with open-source Big Data sets to include these into our analysis and interpretation and to explore potential correlations.
When and how are external evaluations conducted?
We conduct external evaluations either when donors would like to have their project evaluated by an independent third-party evaluator and are willing to finance these, or when we would like to find out more about our own programming and need specific expertise for this that is not to the same extent available inhouse. This is the case for example when we try new program approaches with little or no existing previous research on their impact. Our SAfA Program in Ethiopia is implemented jointly with two other implementing organizations, the DSW and Kühne-Foundation. Since this is the first project of such kind, we are interested in learning if beneficiaries who benefit from activities of all three implementers are better off than beneficiaries who only benefit from activities of one of the implementers. To generate robust evidence of the effects, we hired the renowned research institute IFPRI to conduct an external evaluation of the project using a quasi-experimental evaluation design.
What is best about M&E at HRNS?
We at HRNS M&E are equipped with skilled and dedicated people who are committed to a common goal: to offer the best possible programming to smallholder families we work with to prove ourselves worthy of their time and trust. To achieve this, we aim to build our M&E work on scientific and data-driven approaches, efficient digitalized internal processes and high-quality long-term relationships with farmers and colleagues.
Since we do not see a contradiction here, we M&E people at HRNS aim to combine the best of three worlds: the accuracy, objectivity, and sophistication of scientific academia; the efficiency and drive of the private sector; and altruistic, ethical, and sustainable not-for-profit development work.
M&E enables us tosucceed
Get involved with HRNS
Curious to learn more? Partnering with Hanns R. Neumann Stiftung means to join hands-on and committed work with smallholder families in coffee regions. It means to be willing to bring change for them. It means to be ready to get involved and work together.