These sessions featured a combination of evaluators and data science users and practitioners to explore lessons, opportunities and challenges in rewiring evaluation.  The sessions drew on the new Rockefeller Foundation report, Measuring Results and Impact in the Age of Big Data: the Nexus of Evaluation, Analytics, and Digital Technology, and other lessons from both international development experiences and the ongoing COVID-19 situation.

Watch a re-play the live event

 

Session 1: Overview of Big Data and its Applications to Evaluation (9:00 AM)

The first half of this session will review some of the major challenges facing current evaluation approaches, the ability of big data to address these kinds of problems, and discuss some examples of big data that can address these challenges.

Moderator:

  • Veronica OlazabalSenior Advisor and Director, Measurement, Evaluation and Organizational Performance , Rockefeller Foundation
    Veronica Olazabal, Senior Adviser & Director at The Rockefeller Foundation, is an award-winning evaluator with a global portfolio ranging +15 years. She serves on various funding and advisory boards and has published on this topic in the Evaluation. Under her management, The Rockefeller Foundation recently published Measuring Results and Impact in the Age of Big Data: the Nexus of Evaluation, Analytics, and Digital Technology.

1.1 Evaluation in the Age of Big Data

The presentation will begin by reviewing some of the major challenges facing current evaluation approaches. These include the limited capacity of most evaluation methods: collect the amount and variety of data required to evaluate large and complex programs operating a rapidly changing world; to evaluate complex programs; and to rapidly analyze vast quantities of diverse data. It will then discuss the ability of big data to address these kinds of problems, and will review some of the kinds of big data that can address these challenges, including: geospatial analysis, social media, phone call data records, mobile phones, electronic financial transaction records (ATM etc), the Internet of Things and the huge volumes of underutilized administrative data. Case studies will illustrate some real-world applications of these techniques.

Panelist:

  • Michael Bamberger, Independent Evaluator and Author , Independent
    Michael Bamberger has been involved in development evaluation for over 40 years. Publications include "Integrating big data into the monitoring and evaluation of development programs", "Dealing with complexity in development evaluation", and "RealWorld Evaluation".

1.2 A Big Data Science Approach to Evaluation

This session will present on how big data, statistical modeling and machine learning are coming together to produce more cost-effective, timely, flexible and useful evaluation findings. Audience members will receive an overview of the steps involved in using the tools of data science, including predictive and prescriptive analytics, to build evaluation models that find naturally occurring experiments in historical data to find rigorous counterfactual evidence of what works. The audience will learn about the technology requirements and analytic skills needed for applying this evaluation approach, as well as its benefits, limitations and challenges. 

Panelist:

  • Pete YorkPrincipal and Chief Data Scientist , BCT Partners
    Pete York has over 20 years experience as an evaluation consultant and researcher. For the past eight years, he has built predictive, prescriptive, and causal evaluation models using large administrative datasets and machine-learning algorithms across many fields.

 

Session 2: Organizational Perspectives on Big Data and Evaluation (10:30 AM)

The first part of this session will explore how the Indian context presents unique challenges and solutions to the adoption of data science for evaluation studies in the development sector. Taking into account data collection ethics that the post-COVID-19 era will necessitate, it will further emphasize how innovation and technology can pave the way for greater use of data science for evaluations in India. In the second part, we will explore the availability of new types of data and new techniques for analyzing existing and new data which are generating novel opportunities in the field of evaluation, specifically asking how this trend is influencing the practice of independent evaluation in multilateral (development banks, United Nations) and bilateral agencies in the field of international development. 

Moderator:

  • Veronica OlazabalSenior Advisor and Director, Measurement, Evaluation and Organizational Performance , Rockefeller Foundation
    Veronica Olazabal, Senior Adviser & Director at The Rockefeller Foundation, is an award-winning evaluator with a global portfolio ranging +15 years. She serves on various funding and advisory boards and has published on this topic in the Evaluation. Under her management, The Rockefeller Foundation recently published Measuring Results and Impact in the Age of Big Data: the Nexus of Evaluation, Analytics, and Digital Technology.

2.1 Perspective on Using Data Science for Evaluations - Learning from India

What are the regional challenges to using data science for evaluations in India? This presentation will explore how the Indian context presents unique challenges and solutions to the adoption of data science for evaluation studies in the development sector. Mapping the major sources of big data in India and the larger South Asian narrative around its role in evaluations, the presentation will cover the major gaps and concerns in implementation. Highlighting regional case study examples, how these challenges can be met by designing region-specific solutions will also be addressed. Taking into account data collection ethics that the post-COVID-19 era will necessitate, the presentation will further emphasize how innovation and technology can pave the way for greater use of data science for evaluations in India.  

Panelist:

  • Swapnil ShekharCo-founder and COO , Sambodhi Research and Communications
    Swapnil Shekhar is a co-founder and COO at Sambodhi, South Asia’s leading impact evaluation and research organization in the development and public sector. Swapnil has 20+ years of experience driving large-scale MLE projects across Asia and Africa, being an active contributor to the global south development narrative and field.

2.2 Evaluation in the Age of Big Data: Prospects and Challenges for Independent Evaluation

The availability of new types of data and new techniques for analyzing existing and new data are generating novel opportunities for the field of evaluation. How is this trend influencing the practice of independent evaluation in multilateral (development banks, United Nations) and bilateral agencies in the field of international development? While providing a comprehensive response to this question would be a daunting task, this presentation will discuss some avenues of thought divided into two parts. In the first part, the presentation will discuss some illustrations of particularly promising ways of using big data in combination with machine learning in the context of evaluative work conducted by the Independent Evaluation Group (of the World Bank Group). Specific attention will be paid to aspects of text analytics and the use of geospatial data. In the second part, the presentation will raise the broader question of opportunities and limitations to using big data in the context of independent evaluation. 

Panelist:

  • Jos VaessenMethods Adviser, Independent Evaluation Group , World Bank Group
    Jos is Methods Advisor at the World Bank. Publications include: "Impact evaluations and development – NONIE guidance on impact evaluation", "Mind the gap: perspectives on policy evaluation and the social sciences", "Dealing with complexity in development evaluation: a practical approach".