Field missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and implementation of projects. This ‘reality-check’ stimulates learning and allows the evaluator to fine-tune their questions and methodology.

The current COVID-19 travel restrictions pose significant challenges to field-based assessments of project effectiveness. So, what can evaluators do when they can’t get in the field?

One possibility is to observe project impacts from space. Geospatial data is information collected by satellites pinpointed to an exact geographical location on earth. It is often freely available, covers several time periods, and offers a wide range of interesting indicators. Popular geospatial data are indicators of market accessibility, agroecology, and the environment. A geospatial dataset can thus be constructed by linking multiple geospatial data points with the geographical location of project activities and their surroundings.

The possibility to construct a geospatial dataset for evaluating a project provides a unique opportunity for a robust quantitative assessment of project effectiveness. Beyond effectiveness, geospatial data can also provide a wealth of descriptive information that allows evaluators to better understand the local context. Even if visiting a project site is no longer possible due to COVID-19 related travel restrictions, evaluators can get a detailed picture of what is happening where in the project area by observing from space.

IEG is analyzing geospatial datasets in several of its ongoing evaluations including an urban transport project in Mozambique, a sustainable land- and water-management project in Ethiopia, and a biodiversity project in Madagascar.

Geospatial analysis usually involves two steps. First, geospatial data is used to precisely and accurately measure an indicator of project effectiveness. When a chronological series of geospatial data is available, changes in the indicator can be calculated using different measurements over time. The geospatial data on land use and road infrastructure are of particular interest to IEG’s evaluations. The ‘vegetation greenness’ of the land in Ethiopia is measured by looking at changes over time in the coverage of land with green vegetation. Similarly, deforestation rates in Madagascar are measured as the change in forest coverage of the land surface over time. In Mozambique and India, the density of social and economic activities is measured by the travel distance to urban amenities using roads.

Second, as geospatial information is available for locations beyond the project boundaries, a proper ‘counterfactual’ can be constructed. The counterfactual illustrates a ‘with and without’ scenario - what would have happened at the project location if project activities were not implemented there. Combining the temporal and spatial variation in geospatial data provides a very robust ‘difference-in-difference’ assessment of project effectiveness. The temporal variation identifies the ‘before-and-after’ difference, and the spatial variation identifies the ‘with-and-without’ difference.

The ‘difference-in-difference’ assessment of project effectiveness is applied as follows. In Mozambique and India, IEG compares changes in economic activity between urban areas that were either adjacent to a road improved by the project or adjacent to a nearby but non-improved road. Similarly, long-term changes in vegetation cover in Ethiopia are compared between land parcels in treated watersheds with similar parcels in untreated watersheds within a reasonable distance from the project site. Finally, IEG compares changes in deforestation rates between patches of forests on either side of the border of conservation areas in Madagascar. Then, these changes are compared between conservation areas supported by the World Bank and areas without project support. In each of these scenarios, the analysis informs the broader question of ‘what difference did the project make?’.

Blog_Mumbai.png
     This 3D map shows the changes in the height of the built-up area in Mumbai.

However, not all projects allow for a geospatial analysis of effectiveness. The availability of geospatial data to measure project indicators depends on the sector, the type of project, and the nature of activities. Projects without a specific geographic location, such as projects supporting a development policy at the national level, do not lend themselves to a geospatial analysis.

But even if a quantitative geospatial analysis is possible, asking whether a project was effective might not be the most important question for the project evaluation. The more interesting evaluation questions are often those looking at the factors limiting the project’s impact. These factors are often highly contextual and linked with human behavior, which is much more difficult to measure from space. So, the quantitative geospatial analysis is an important first step to assess project effectiveness, but evaluations need to go further and understand why the project has been effective or not.

But geospatial data can have an important contribution here as well. The geospatial information on contextual factors, such as the cover of the land or travel time to reach a given location, can help to identify different levels of project effectiveness and understand the role of underlying drivers in explaining the observed differences. In a follow-up blog, we will elaborate on how geospatial analysis can guide the design of a qualitative data collection method.

Pictured at the top of the page: This image displays the change in forest cover of the land surface in Madagascar from 1990 to 2017. The black line represents the boundary of two Protected Areas for biodiversity conservation in Madagascar, Manongarivo and Tsaratanana. The blue dotted line is the 5 km buffer around the border of the Protected Area. Green dots are land that remained forest over time, white dots are land covered with forests, and red dots are land that were deforested during the period 1990 to 2017. IEG analyzed the share of different dots on each side of the border of Protected Areas to assess deforestation rates in and around Protected Areas.

Comments

Submitted by Serdar Bayryyev on Mon, 11/16/2020 - 09:39

Permalink

Indeed, there is a huge potential for using modern information technologies for evaluation and measurement of development results. To support the illustration of how geospatial data can be used to that extent, I would like to share the example of using modern remote sensing tools in evaluation work.
The evaluation of the large, multi-million dollar irrigation rehabilitation programme in Afghanistan has benefitted from the use of the tool that enables data collection through Google Earth. This open-access data tool, called Open Foris Collect Earth (http://www.openforis.org/tools/collect-earth.html), developed by the UN’s Food and Agriculture Organization (FAO), allows potential users to track land-use and landscape changes anywhere in the world with the help of satellite imagery. FAO’s Open Foris Collect Earth tool is web-based, free of charge, requires no downloads or installation, and allows users to assess relevant data on any location on the Earth with satellite data. It can be accessed by clicking on a link (https://collect.earth/) and registering on the platform.

While the main purpose of the new platform is to support countries to assess and monitor forest cover, and assist in gathering, producing and disseminating reliable information on the state of forest resources, our experience showed that it can also be potentially used for evaluation of development programmes. In the course of the evaluation of the UN’s FAO Programme for Improvement of Irrigation Systems in Afghanistan, we faced serious limitations in accessing project sites due to security and other constraints (e.g. weather conditions, length of irrigation network, etc.). We have therefore opted for using the Collect Earth tool for remote assessment of the irrigation network expansion and of the intensity of agricultural activity in areas with newly restored irrigation. The tool allowed us to assess whether specific plots of irrigated lands have achieved double cropping, single cropping or had no cropping at all, based on the satellite images of the vegetative cover. This function enabled us to compare most recent data on the expansion of the irrigations canals and the vegetative cover along sections of the rehabilitated canals with the historical baselines. More details on this evaluation can be found in the following link: http://www.fao.org/3/a-bd685e.pdf.

Kind regards,
Serdar Bayryyev
Senior Evaluation Officer
UN Food and Agriculture Organization

Add new comment