As the implications of the Covid-19 pandemic are felt around the world, few will escape its economic repercussions.  Competition for financial resources will be fierce, not only between developing economies but between developing and industrial economies. Bolstering the  ability of governments to maintain domestic support for global efforts to respond to the needs of the most vulnerable in the midst of this crisis will be critical.

Institutions like the World Bank Group (WBG), as they seek to respond to COVID-19 impacts, as well as development needs more broadly, will need to assure taxpayers and voters in donor countries that increasingly scarce resources are being well spent.  This will require creativity in adapting monitoring and evaluation (M&E) systems to match both the speed and forms of the World Bank Group’s crisis response and its continuing pursuit of its development mandate.

Applying Lessons from M&E in Insecure Areas to COVID-19

Infographic about digital data collection with tips from the blog.png
         Click to see a larger version.

World Bank teams with experience working in “no-go” areas with high risks to personal security have much to teach about M&E in the face of COVID-19.

In situations of conflict, one of the ways in which M&E (and data collection in particular) has been carried out has been by drawing on Third-Party Monitoring. However as discussed by Lauren Kelly and Marie Gaarder in a previous IEG blog, there are ethical considerations associated with sending people into harm’s way.  These have been  exacerbated  by COVID, and must be taken into account in M&E efforts.  This is because there is significant risk of third-party monitors both spreading and contracting the disease, including between densely populated capitals and more remote, yet unaffected areas. These risks are further compounded by inadequate supplies of protective equipment and a lack of access to COVID-19 testing in many of the countries in which the World Bank operates. 

To address these risks, Elias Sagmeister of the Global Public Policy Institute provides a useful toolkit of tech-enabled handheld devices for digital data collection, including mobile phone-based feedback mechanisms, remote sensing with satellites or delivery tracking, communication with online platforms and broadcasting with radios and other forms of (social) media.  But technology has its limitations.

  • First, technology tools need to be designed in line with data collection requirements. Many tech enabled tools are best used to verify assets; they are less equipped to measure quality and implementation fidelity.
  • Second, they work best as “a bundle”: earth observation technologies paired with phone-based mechanisms can help verify that assets were delivered and to where, as well as to whom, they were intended to be delivered.
  • Third, one needs to be mindful of exclusionary factors surrounding the use of technology. In insecure areas, women often have less access to cell phones than men and this can bias reported outcomes, as was shown in IEG’s radio and mobile-based outreach in the Afghanistan Country Program Evaluation.
  • Fourth, data collected by tech enabled tools are unlikely to be able to capture the granularity needed to ascertain and address critical environmental, social and conflict related risks, that are more adequately identified through consultations, rigorous supervision, and implementation of citizen engagement processes.

 

Lessons Also Can be Gleaned from the Ebola Experience

If there is one thing we have learned from Ebola, it is that communication matters. In the times of COVID-19, like with Ebola, there will be heightened fear, and confusion, and distrust of governments and donors who are “trying to do the right thing” by conveying hard messages about health and behavior to isolated, disbelieving communities.  

M&E has a critical role to play during a pandemic in assessing the continued appropriateness of World Bank activities given the enhanced need to protect personal safety. Do current projects require citizens to participate  in-person, in unsafe ways to achieve intended benefits?  Are projects helping clients to adapt these systems – education, health services, labor,  payments -  to maintain essential services and income - while protecting citizen safety? And how will we know?

From a communications standpoint, one of the key lessons from West Africa is that responders’ unfamiliarity with a local culture can undermine pandemic response.  “Foreigners working for the UN, the Red Cross and other international organizations were not necessarily the most effective communicators on Ebola.” This is because when people’s lives are threatened, they want to hear from those closest to them—those who speak their language.  M&E has a role to play in helping people and projects to adjust modalities alongside the COVID realities. To achieve this, both tech-enabled and human data collection and evaluation efforts need to be culturally and linguistically  sensitive.   

Countries providing the essential resources for the pandemic response will want to know how funds released through a project’s “Contingency Emergency Response Components” are being spent, but project M&E frameworks are often not equipped to do this. Restructured, adaptive M&E and safe data gathering methods need to be developed alongside the release of such  funds, in real time. 

This will be particularly important as the global community commits vast sums to support developing countries in responding to COVID-19, including the recently approved $14 billion package of fast-track financing initiative from the World Bank Group to assist companies and countries in their efforts to prevent, detect and respond to the rapid spread of COVID-19.  The challenge is to ensure that M&E is not forgotten in the well intentioned rush to respond to urgent needs.

 

Do you have ideas, lessons, or insights to contribute to this conversation about evaluation during COVID-19?  Please share them in the comments below.

 

photo credit: Shutterstock/ Ms.Lotus Bua

Comments

Submitted by Chris Nelson on Wed, 04/01/2020 - 14:35

Permalink

Thanks Lauren and Jeff, I think a key message is to be proactive about just collecting any data when responding to emergencies. In the rush to deliver something, we forget to monitor and reflect as we go. Answering questions at the end becomes impossible without knowing why decisions were made at the time. Using and capturing data at various decision points along the way helps us to know why we did what we did at the time. The United States military have become good at this through their emergent learning approaches (including before-action-reviews and after-action-reviews). Formalizing these simplistic steps and capturing the interactions occurring in emergency situations could be useful to a better response to future pandemics.

Submitted by MD. EHTESHAM, … on Thu, 04/02/2020 - 06:00

Permalink

Thank you so much for great insights about carrying out M&E work at the time of crisis, when human intervention may be risk. I have some experience on participatory performance monitoring, where we identify and train some selected community based men, women, adolescents or any change agents. These individuals provide data and information on the a range of issues and technical experts sitting in office, use it as per the programme indicators.

Yes, this seems difficult but lot of energy we require to invest to make the selected individual in community trained and the questions are designed in such a way that they feel comfortable responding. As far as platform is concerned, Mobile phone calls, whatsApp messages, audio / video call works well in this situation.

If you compare it with the in person visit to community and collect data and information, it is not comparable however, both model has their own Pros and cons.
Considering the time like COVID-19, if you have already prepared your community on this model, it will work and for sustainability point of view also, it has huge scope.

I would like more to learn on this aspects if somebody has worked on this.

Stay Home- Stay safe- Pray for all Mankind !

Thanks
Ehtesham

Submitted by Arlette Pichar… on Thu, 04/02/2020 - 14:09

Permalink

Gracias por tan inspiradoras ideas. Hoy más que nunca se requiere desarrollar la creatividad y propiciar el trabajo colaborativo.
Comentarles que estamos trabajando en una propuesta de Evaluación del Impacto del Covid-19 (desde una perspectiva muldimensional y un enfoque integral), iniciativa que hace unos días lanzamos a las comunidades de evaluadores de América Latina y el Caribe.

Submitted by Rabindra Suwal on Tue, 04/07/2020 - 23:29

Permalink

Well mention: "The challenge is to ensure that M&E is not forgotten in the well intentioned rush to respond to urgent needs." Personally, I have been watching the data generation, data analysis, data presentation and data open in regard of COVID19 response of US, India and Nepal (My country). I found US is working excellently in all aspect of data. But both country India and Nepal are just making the number counting also lack access of detail data for public or for interested M&E practitioner. I have started to compile Nepal data that generates from official channel as well as from different public media such as online news portal and shared to M&E professional network . So independent M&E professional/researcher would not stuck with no insightful data for now and future. Here's you can have a look https://drive.google.com/open?id=19rBi4mxvm9hN6uOnSQxC37hYvsm8V_q7

Submitted by Gustavo Biasol… on Wed, 05/06/2020 - 10:36

Permalink

Obrigado pelo texto!
Muito provocativo e preciso!
Um ponto fundamental é prover os agentes públicos com dados precisos sobre a situação em cada país, o que é impossível sem a realização de um número adequado e padronizado de testes por milhão de habitantes. Creio que esta deva ser uma diretriz que o Banco Mundial deva apoiar.

Thank you for the text!
It is very accurate!
One key point is to provide public agents with precise data what is imposible without an accurate and standartizes number of tests per milliion of inhabitants. So I believe this is a way World Bank should support.

Submitted by Minal Mehta M… on Tue, 07/14/2020 - 03:23

Permalink

My experience in M&E from the previous job has helped me. In private practice, I am associated with one group that is a collective of health care providers who provide free telemedicine. I have got calls from clients in severe stress, phobia & panic about the disease and suicidal ideation. The immediate need is to reassure share facts and engage them in monitoring the community. Telling them to check if the ten people in a day can be reached to ensure that they follow good practices - steam inhalation, drinking herbal tea, breathing exercises, hand hygiene, social distancing, and mask-wearing. Giving a timeline that is suitable for the client -usually 24 hrs.

Once they report back on the good practice indicators of monitoring they are acknowledged as to how important it is to have them in the society and that they need not consider suicide but participate in tele supporting /monitoring and contribute. In the small capacity, I found engaging people in COVID 19 monitoring a very rewarding experience. Engaging CORONA survivor in pandemic monitoring will be a very simple solution to increased partnership and community ownership

Add new comment