Behind the Numbers: Feed the Future’s New Impact Data Explained

Have you read the 2016 Feed the Future Progress Report yet? Thisyear, it includes new data on changes in poverty and stunting in nearly all the developing countries where Feed the Future targets its food security and nutrition efforts.

These data show tremendous gains toward our goals to reduce poverty and stunting. Where did these numbers come from and what can we interpret from them? We asked Feed the Future’s monitoring, evaluation and learning lead, Emily Hogue, to explain.

Keep reading for more on the story these numbers tell — and the story behind them.

---

How did you get these impact numbers?

We used internationally-recognized and peer-reviewed approaches for measuring poverty and stunting to calculate the impact values from population-based survey data.

First, we collected baseline survey data to determine the prevalence of poverty and prevalence of stunting in our Feed the Future target geographies (regions or “zones of influence” within our focus countries). These set a first (baseline) data point to which we could compare values in future years once our programs had time to take root and make a difference. When available, we used data from existing secondary sources from nationally-supported surveys, like the Demographic and Health Surveys or the Living Standard Measurement Study surveys, and analyzed the data records that corresponded only to our zones of influence to calculate the value. In countries where surveys weren’t collected or weren’t available for the time frame we needed (2014 to 2016) for a second (interim) data point, or where the sample size wasn’t large enough to provide the needed statistical precision, we collected our own surveys in the zone of influence. The impact numbers represent the difference between poverty and stunting from baseline to interim data. There will be updates to these impact numbers in 2017-2018 after the third data collection point, which will enable us to assess whether we have achieved our five-year targets on poverty and stunting. You can find more details about our approach to collect and analyze impact data in Volume 11 of our Feed the Future Monitoring & Evaluation Guidance Series.

We’ll publish a data set that provides country-by country Feed the Future annual results for output and outcome indicators in the coming year. Before we can share the entire data sets for impact-level indicators, we need to assess and review the data against full survey reports for each country and receive approval from the partner government to publicly release them.

What can we discern from these numbers?

The interim numbers provide our first chance to see the real impacts of our efforts specifically in the Feed the Future zones of influence across nearly all of our focus countries. They demonstrate that our approach through Feed the Future is stimulating the type of transformation needed to improve people’s lives.

At the same time, in some cases these data are showing us where our efforts haven’t yet led to the change we want to see. While not as encouraging, these cases help us learn where we need to intensify our efforts or change our approach to reach our goals. In those countries where topline indicators don’t show we are on track to reach our goals, we are digging deeper into the available data to make sense of what is happening. We’re looking at what other national data and reports are saying, any available academic papers, other data sets, and what our implementing partners are saying in their annual progress reports to inform solutions to any identified problems.

As you read through this year’s progress report, you’ll notice that some of the numbers in the report are directly attributable to Feed the Future programs, and some aren’t. We try to make the distinction very clear by delineating between results and impact. The output and outcome data in the progress report (like number of farmers reached, value of incremental sales, number of children reached, etc.) are attributable to U.S Government programs. These types of results, combined with host-country and other development efforts, are contributing to early impact. The impact numbers we report reflect the status or changes in poverty and stunting in areas where we are intensely working to increase agricultural growth and improve nutrition. Changes in those numbers are not directly attributable to our interventions, but given the breadth and magnitude of efforts, we believe our interventions are certainly contributing to them.

What can’t we tell from the impact numbers?

While changes in the interim values for our zones of influence cannot be directly or rigorously compared to previous trends or changes in national-level values to quantify the magnitude of our impact, they can help us understand if progress in the specific geographic areas where we are concentrating our efforts appears to be accelerating. We designed our Feed the Future country strategies to capitalize on opportunities and chose to work in specific geographical areas where we could collaborate with other partners. This made our zones of influence distinct from other parts of our focus countries and, today, makes it more difficult to distinguish and control for the various factors and influences you need to measure in an impact evaluation.

Numerous factors may affect the rates of change in poverty and stunting, and we can’t assume that any acceleration is solely due to our work. Making those comparisons requires a broader examination of evidence to understand just how much change we are generating through Feed the Future’s efforts. By looking at numerous data sources we can examine what contributions we are making in the zone, what results or outcomes we are causing, and assess whether our interventions could plausibly be contributing to the changes in poverty and stunting.

So, is Feed the Future on track to meet its goals?

The interim data show tremendous progress toward Feed the Future’s goals of reducing poverty and stunting each by an average of 20 percent over five years in the areas where we work. In some countries, Feed the Future is on track to exceed these targets if momentum is maintained. At the same time, there are some countries where we aren’t seeing large changes or where thelevel of change cannot be precisely assessed because the sample size was not large enough.

There are several reasons for this:

• Stunting and poverty are difficult development problems to address and slow to change. Small changes can be very hard to detect without extremely large sample sizes. The interim data are tracking the change in just three or so years of time. In some cases, the survey samples were not large enough to capture small changes, and the margin of error was just too great to conclusively demonstrate change.
• Change often isn’t linear, so while the interim data point may be “midway” in terms of timing, it may not be for the trajectory toward our five-year goal. We’ll establish a third data point that can tell us more in the coming years.
• Various economic, social, political, and environmental factors are hindering progress.  We know that El Niño-induced droughts, regional economic stagnation, and even programmatic delays have impeded progress toward our goals in some countries.

However, the steady increases we are seeing in Feed the Future’s results, coupled with very encouraging reductions in poverty and stunting in many of our focus countries, give us cause to be more than cautiously optimistic that we will meet, and in some cases exceed, our five-year goals.

What’s next?

Measuring these very high-level impacts through repeated surveys is a hallmark approach of Feed the Future’s rigorous accountability framework, and many development leaders and other organizations have recognized the value of this approach for clearly demonstrating what the initiative has achieved. Feed the Future’s monitoring and evaluation approach enables us to capture our contribution to impacts at a population-level, beyond outputs and outcomes. As we move forward to a new phase under the U.S. Government Global Food Security Strategy, we are reflecting on the strengths of our monitoring and evaluation approach as well as its challenges to make our accountability and learning efforts even more robust.

We’ll also be applying findings from the Feed the Future Global Performance Evaluation that is nearing completion and will soon be publicly available. In 2017, we will be working on developing a revised set of indicators and monitoring processes that apply to the U.S. Government Global Food Security Strategy’s revised results framework and theory of change. We anticipate that we will still track our impacts through population-based survey approaches, but some of the indicators and the frequency with which we collect data or report on them may change. Additionally, we will revisit our approach to setting goal-level targets and revise our Feed the Future Learning Agenda.

We have busy months ahead, but look forward to the opportunity to upgrade our approach to accountability and learning so that our programs can improve the lives and food security of increasingly more people around the world.

Read More:

• Check out the data in Feed the Future’s 2016 progress report
• View all the volumes in Feed the Future’s M&E Guidance Series

Editor’s Note: The author is the division chief for monitoring, learning and evaluation in USAID’s Bureau for Food Security.