Search

Displaying 1 - 10 of 3011

Investing in Evaluation Capacity Development in India: Why it Matters Now More than Ever

Web Resource
Investing in Evaluation Capacity Development in India: Why it Matters Now More than Ever
Download the infographic Download the infographic

Investing in evaluation capacity development in India: Why it matters now more than ever

Web Resource
Investing in evaluation capacity development in India- why it matters now more than ever
Governments around the world face the daunting task of addressing a downward spiral of economic activity coupled with a growing health burden from the spread of the coronavirus disease. India is no exception, and the government has had to be innovative in both designing policies and deploying resources to cope with the twin challenges. We believe that two elements can be game-changers in Show MoreGovernments around the world face the daunting task of addressing a downward spiral of economic activity coupled with a growing health burden from the spread of the coronavirus disease. India is no exception, and the government has had to be innovative in both designing policies and deploying resources to cope with the twin challenges. We believe that two elements can be game-changers in addressing the crisis: the use of data and leveraging partnerships. For almost a decade, CLEAR South Asia has been collaborating with state governments in India to undertake systematic capacity-building efforts on data and evidence use for policy decision-making.  Policymakers at the state and central levels have to lead the charge on developing data capacity and building strategic partnerships. They will need to think innovatively to respond swiftly to emerging challenges. It has been gratifying to see the government’s creative use of real-time data and technology for planning containment strategies and service delivery.  We are also seeing an exciting mix of organizations, such as technology enablers, private companies, non-profits and research institutions complementing government efforts to tackle the most difficult challenges and protect the most vulnerable. For these opportunities to be genuinely advantageous, the government must be able to collate and analyze data from multiple sources to understand fully the nature of problems confronting us and to respond effectively. Strengthened data capabilities of the government, whether independently or by leveraging partnerships, to interpret, absorb, and use data and evidence to make informed decisions are urgently needed. However, system-level changes take time, and data use capabilities cannot be built overnight. The foundation of a systems change that is conducive and incentive-compatible for governments to internalize a data-driven approach needs to be laid in advance.            Enlarge and download infographic Creating sustainable channels Leveraging our host institution J-PAL SA’s institutional partnership (now in its sixth year) with the state government of Tamil Nadu in India, CLEAR SA, in collaboration with the state bureaucracy, has developed and executed a multi-pronged, customized capacity-building strategy.  We engage with multiple levels of government to build capabilities across domains, using customized workshops, hands-on training, and advisory. These efforts have culminated in structural channels that allow for useful feedback loops to inform decisions.  Our capacity-building approach had three key features: First, our long-term, government-wide partnership in Tamil Nadu is founded on a 360 degree, deeply embedded life-cycle approach which forges linkages between research, capacity building, and policy advisory to enable data use for decisions. Second, we now know that knowledge transfer is most effective when combined with live examples. Adoption is greater when demonstrated and allows for learning by doing. Multiple touchpoints and continued engagement have helped build trust and value, and sustain the commitment through elections, transfers of key personnel, and shifts in policy priorities. Third, we recognize that decision-making, especially on the adoption of new ideas or practices can be non-sequential– meaning that what we build and recommend today, could come to use a year or more later. A key strength is in being able to identify and be responsive to an opportunity whenever it emerges. In our experience, this is possible when the groundwork is laid upfront. Thus, when a policy window opens, the only incremental effort needed is to refresh and connect the dots, and not have to start from scratch. A longer-term systems-driven capacity-building approach can lead to increased sensitivity and reception to data-driven decision-making among governments. Tamil Nadu is a good example, which has a substantial aging population, and their well-being is an important priority. The Department of Economics and Statistics, in collaboration with leading researchers, launched the first-ever state-wide elderly panel survey in 2016-17.  For almost four years, CLEAR SA provided technical advisory services and training workshops (mirroring activities on the project timeline) on sampling, questionnaire design, and data quality to enable rigorous and efficient data collection. Our capacity-building efforts led to the adoption of independent backchecks (a standard practice in research) by the department to ensure the quality of data.  Last year, the department completed the baseline survey across five districts. Taking cognizance of a critical finding of a growing proportion of elderly living alone, the government announced in their latest budget, a pilot intervention of elderly daycare centers in the state. In addition to these policy wins, the department also adopted digital data collection for their surveys and plan to conduct a follow-up wave next year. We are now planning an evaluation of this pilot to inform expansion approaches. Further, during the current crisis, via the use of phone surveys, the government and researchers can track whether the elderly covered in the survey are facing any issues during the Covid-19 lockdown.   Such a holistic, long-term, and embedded approach was instrumental in creating systems for new and high-quality data collection, use of data for planning, decisions, and increased appetite for further evaluations. It also means that the goal of adopting a systematic approach to designing innovative policies and deploying resources to protect a vulnerable population such as the elderly, is achievable. Photo credit Shutterstock/ By Myvector

When evaluators cannot make it to the field, they can always observe from space

Web Resource
change in forest cover of the land surface in Madagascar from 1990 to 2017.
Field missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and Show MoreField missions are at the very core of project evaluation. An evaluator will start with a desk-based review of available project information and prepare a methodology to assess the effectiveness of a project. However, it is only by interacting with policymakers, implementing agencies, and project beneficiaries that the evaluator gets a better understanding of the reality affecting the design and implementation of projects. This ‘reality-check’ stimulates learning and allows the evaluator to fine-tune their questions and methodology. The current COVID-19 travel restrictions pose significant challenges to field-based assessments of project effectiveness. So, what can evaluators do when they can’t get in the field? One possibility is to observe project impacts from space. Geospatial data is information collected by satellites pinpointed to an exact geographical location on earth. It is often freely available, covers several time periods, and offers a wide range of interesting indicators. Popular geospatial data are indicators of market accessibility, agroecology, and the environment. A geospatial dataset can thus be constructed by linking multiple geospatial data points with the geographical location of project activities and their surroundings. The possibility to construct a geospatial dataset for evaluating a project provides a unique opportunity for a robust quantitative assessment of project effectiveness. Beyond effectiveness, geospatial data can also provide a wealth of descriptive information that allows evaluators to better understand the local context. Even if visiting a project site is no longer possible due to COVID-19 related travel restrictions, evaluators can get a detailed picture of what is happening where in the project area by observing from space. IEG is analyzing geospatial datasets in several of its ongoing evaluations including an urban transport project in Mozambique, a sustainable land- and water-management project in Ethiopia, and a biodiversity project in Madagascar. Geospatial analysis usually involves two steps. First, geospatial data is used to precisely and accurately measure an indicator of project effectiveness. When a chronological series of geospatial data is available, changes in the indicator can be calculated using different measurements over time. The geospatial data on land use and road infrastructure are of particular interest to IEG’s evaluations. The ‘vegetation greenness’ of the land in Ethiopia is measured by looking at changes over time in the coverage of land with green vegetation. Similarly, deforestation rates in Madagascar are measured as the change in forest coverage of the land surface over time. In Mozambique and India, the density of social and economic activities is measured by the travel distance to urban amenities using roads. Second, as geospatial information is available for locations beyond the project boundaries, a proper ‘counterfactual’ can be constructed. The counterfactual illustrates a ‘with and without’ scenario - what would have happened at the project location if project activities were not implemented there. Combining the temporal and spatial variation in geospatial data provides a very robust ‘difference-in-difference’ assessment of project effectiveness. The temporal variation identifies the ‘before-and-after’ difference, and the spatial variation identifies the ‘with-and-without’ difference. The ‘difference-in-difference’ assessment of project effectiveness is applied as follows. In Mozambique and India, IEG compares changes in economic activity between urban areas that were either adjacent to a road improved by the project or adjacent to a nearby but non-improved road. Similarly, long-term changes in vegetation cover in Ethiopia are compared between land parcels in treated watersheds with similar parcels in untreated watersheds within a reasonable distance from the project site. Finally, IEG compares changes in deforestation rates between patches of forests on either side of the border of conservation areas in Madagascar. Then, these changes are compared between conservation areas supported by the World Bank and areas without project support. In each of these scenarios, the analysis informs the broader question of ‘what difference did the project make?’.      This 3D map shows the changes in the height of the built-up area in Mumbai. However, not all projects allow for a geospatial analysis of effectiveness. The availability of geospatial data to measure project indicators depends on the sector, the type of project, and the nature of activities. Projects without a specific geographic location, such as projects supporting a development policy at the national level, do not lend themselves to a geospatial analysis. But even if a quantitative geospatial analysis is possible, asking whether a project was effective might not be the most important question for the project evaluation. The more interesting evaluation questions are often those looking at the factors limiting the project’s impact. These factors are often highly contextual and linked with human behavior, which is much more difficult to measure from space. So, the quantitative geospatial analysis is an important first step to assess project effectiveness, but evaluations need to go further and understand why the project has been effective or not. But geospatial data can have an important contribution here as well. The geospatial information on contextual factors, such as the cover of the land or travel time to reach a given location, can help to identify different levels of project effectiveness and understand the role of underlying drivers in explaining the observed differences. In a follow-up blog, we will elaborate on how geospatial analysis can guide the design of a qualitative data collection method. Pictured at the top of the page: This image displays the change in forest cover of the land surface in Madagascar from 1990 to 2017. The black line represents the boundary of two Protected Areas for biodiversity conservation in Madagascar, Manongarivo and Tsaratanana. The blue dotted line is the 5 km buffer around the border of the Protected Area. Green dots are land that remained forest over time, white dots are land covered with forests, and red dots are land that were deforested during the period 1990 to 2017. IEG analyzed the share of different dots on each side of the border of Protected Areas to assess deforestation rates in and around Protected Areas.

Keeping the Private Sector Alive During the Coronavirus (COVID-19): 5 lessons from past crises

Web Resource
Keeping the Private Sector Alive During the Coronavirus (COVID-19): 5 lessons from past crises
Click to download the infographic Click to download the infographic

Lessons from Evaluation: Support and Financing to the Formal Private Sector in Response to COVID-19

PDF file
This note identifies core lessons for the Bank Group on addressing the impact of the crisis on business and enterprises, based on evaluative evidence from the Independent Evaluation Group (IEG). It particularly draws on Bank Group experiences in addressing earlier crises, including the global economic crisis of 2008–10, the food crisis of 2007–8, and the East Asian crisis of 1998. It also reviews Show MoreThis note identifies core lessons for the Bank Group on addressing the impact of the crisis on business and enterprises, based on evaluative evidence from the Independent Evaluation Group (IEG). It particularly draws on Bank Group experiences in addressing earlier crises, including the global economic crisis of 2008–10, the food crisis of 2007–8, and the East Asian crisis of 1998. It also reviews evidence from responses to other systemic shocks, such as natural disasters and crises arising from conflict. However, it does not reinterpret past findings in light of subsequent developments. Lastly, it incorporates IEG’s broader evaluative findings on instruments that support business and market development. It complements other IEG notes on crisis response topics under preparation, including those on distressed assets and trade finance.

Keeping the private sector alive during the coronavirus (COVID-19): 4 lessons from past crises

Web Resource
Corona virus economic impact concept image
How can the World Bank Group help keep the formal private sector alive during the current coronavirus (COVID-19) crisis?  Beyond its impact on public health, efforts to limit the spread of COVID-19 are taking a toll, damaging businesses and livelihoods across the world. Trade and transport are disrupted, many businesses are idle, and workers and households have lost jobs and income.  By Show MoreHow can the World Bank Group help keep the formal private sector alive during the current coronavirus (COVID-19) crisis?  Beyond its impact on public health, efforts to limit the spread of COVID-19 are taking a toll, damaging businesses and livelihoods across the world. Trade and transport are disrupted, many businesses are idle, and workers and households have lost jobs and income.  By providing timely and effective support and financing, development agencies can help the formal private sector survive. Here’s what the World Bank Group’s experience in earlier crises tells us.  As international financial institutions look to help the private sector cope with the economic shocks of the coronavirus pandemic, past global crises offer valuable lessons on what works.   At the Independent Evaluation Group (IEG), we have mined our evaluations of World Bank Group responses to a range of global crises, alongside assessments of programs to support the private sector, and identified four overarching lessons to guide efforts to help businesses survive the impacts of the coronavirus . In summary, the lessons suggest a need to find ways to act fast to support the private sector, to ensure that assistance reaches those enterprises in distress, to build on prior knowledge of business conditions and constraints, and to understand that restoration of growth and employment requires a sustained response.  1) Businesses need help quickly, so international financial institutions must act fast Governments are often the fastest way to get support to the private sector. The World Bank’s Development Policy Loans (DPL) provide general budget financing to governments to allow spending to address the crisis and fill crisis-induced revenue gaps.  Budget support allows governments to channel resources to banks and businesses to fund payrolls, provide guarantees, credit or loan forbearance to help firms survive although they cannot produce or sell. World Bank Investment Lending can get to enterprises faster when adding finance to existing loans and when designing new simple or repeater loans. The International Finance Corporation (IFC) of the World Bank Group, which is already engaged with private banks and businesses, can respond more rapidly to keep the private sector alive when it focuses on programs and instruments that already have a solid track record and have shown the capacity for rapid mobilization during a crisis. Following the 2008 global financial crisis, the IFC launched several new initiatives to support businesses but their set-up time and the lags in implementation limited their short-term impact.  On the other hand, IFC’s Global Trade Finance Program , an existing facility, was able to increase its support for trade finance and reach out to new banks.  New instruments may be more appropriate for the medium term.  Click to enlarge and download the infographic 2) Make sure projects reach the businesses that need the help Rapid project preparation is critical during a crisis, but it is vital that projects are designed with effective systems for targeting the hardest hit firms and monitoring to ensure the help has actually reached them. The primary aim of most crisis-related World Bank Group financial intermediary loans (FILs), was to increase bank credit for private sector groups most affected by the crisis, such as small and medium enterprises, exporters needing trade finance, rural businesses, and cooperatives.  FILS have been widely used during crisis -- including after the 2008 crisis. Subsequent evaluations found that few FILs were able to disburse rapidly, targeting was an ongoing problem for many of them and the monitoring of the impact of the crisis financing component was weak and often not reported.  Reaching micro, small and medium enterprises poses additional challenges due to their limited size and bargaining power. Besides loans, matching grants can be helpful and business development services appear to help improve firm performance and create jobs.  Yet a better understanding of how they work and how they can be used to respond to crisis is needed. Partial credit guarantees that cover a share of the default risk of loans can also help, but their effectiveness depends on the strength of a country’s legal and regulatory frameworks.   3)  Understanding the business environment is key to helping businesses Drawing from an existing stock of knowledge or carrying out new analytic and advisory work can ensure that interventions are aimed at the most important problems faced by the private sector, and that resources are directed to their best use. During the 2008 global financial crisis, earlier analytical work provided a platform for the World Bank’s response (and sometimes that of other donors as well). In situations of fragility and conflict, Risk and Resilience Assessments (RRAs) can complement private sector diagnostics and help guide interventions that both support businesses and address the drivers of instability.  In countries where pre-crisis engagement was low, knowledge gaps left the Bank unprepared to help map out actionable, forward-looking programs and the quality of lending suffered. 4) When the crisis is over, the private sector still needs support Even when responding to a crisis, there is a need for longer term planning. This should be focused on an enduring restoration of growth and employment, and sustained responses. A strategic roadmap for crisis engagement, that sequences interventions from short term to longer term can be beneficial. Such a roadmap for crisis engagement should be based on ongoing, systemic analysis of stress factors, a framework for coordination within the World Bank Group and with other international financial institutions, and a review of instruments for effective crisis support, meaningful growth and medium-term development. For more details, please see the learning note that elaborates on each of the lessons. Please visit the IEG Lessons Library for a range of resources relevant to the COVID-19 response   Image credit: Shutterstock/ ffikretow

Meet the Evaluator: Lauren Kelly

Web Resource
Meet the Evaluator: Lauren Kelly
The Independent Evaluation Group’s Lauren Kelly speaks on the role of the evaluator during the ongoing pandemic – and infodemic. The Independent Evaluation Group’s Lauren Kelly speaks on the role of the evaluator during the ongoing pandemic – and infodemic.

Building the First-Ever Partnership Focused on Addressing Global Gaps in Evaluation Capacity

Web Resource
Building the First-Ever Partnership Focused on Addressing Global Gaps in Evaluation Capacity
With the ten-year countdown to the Sustainable Development Goals (SDGs) now underway, and countries across the globe struggling to cope with the COVID-19 pandemic, the capacity to gather data to inform decisions, and to monitor and evaluate the impact of policies, is now an urgent priority. A broad coalition of governments and national and international organizations have agreed to establish the Show MoreWith the ten-year countdown to the Sustainable Development Goals (SDGs) now underway, and countries across the globe struggling to cope with the COVID-19 pandemic, the capacity to gather data to inform decisions, and to monitor and evaluate the impact of policies, is now an urgent priority. A broad coalition of governments and national and international organizations have agreed to establish the first-ever global partnership focused on addressing the worldwide gaps in monitoring and evaluation capacity. In June 2020, a range of donor countries and organizations met for the first time in a Co-Creation Workshop to discuss concrete steps towards establishing an inclusive partnership to meet the global demand from developing countries for stronger monitoring and evaluation systems and capacity. The aim of the partnership is to increase coordination for greater impact among the various national and international initiatives aimed at building evaluation capacity, and to pool resources and draw on local and global expertise and knowledge to scale up these efforts. The workshop was a three-day virtual brainstorming discussion focused on building consensus around a joint vision of the global partnership. Ahead of the workshop, a series of consultations were held with representatives from countries committed to strengthening their M&E systems and capacities, to understand the challenges they face, and how best to support their programs. {"preview_thumbnail":"/sites/default/files/Data/styles/video_embed_wysiwyg_preview/public/video_thumbnails/ftcYucMsIlE.jpg?itok=y3Fn6dDT","video_url":"https://youtu.be/ftcYucMsIlE","settings":{"responsive":0,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (854x480)."]} Effective monitoring and evaluation systems are an essential ingredient for advancing the sustainable development goals as they foster accountability and evidence-based policy making. This innovative partnership will take us a step closer in addressing the worldwide demand from countries for stronger M&E systems and capacities for more inclusive and sustainable development results. Oscar A. Garcia, Director, Independent Evaluation Office, UNDP The current demand for evaluation capacity development far outstrips the resources and reach of any single institution, and the impact of the many programs launched to meet this need is diluted by a lack of coordination. Earlier this year, IEG and the UNDP’s Independent Evaluation Office (IEO) signed an agreement on closer collaboration on meeting this need. Recently, IEG also signed an agreement  with Canada’s École nationale d'administration publique (ENAP) in order to coordinate actions and pool expertise and resources towards meeting the need for stronger M&E systems and capacity in key, under-served regions of the world. The lack of robust monitoring and evaluation systems leaves many countries at a disadvantage and has become an ever more urgent development challenge in the face of the fast-moving coronavirus pandemic. Only by working together will we be able to address the global gaps in evaluation capacity, and ensure no communities or countries are left behind. Alison Evans, World Bank Vice President and IEG Director-General Along with developing a joint vision, the participants in the Co-Creation Workshop also discussed the key lines of business and activities in providing countries support on strengthening their monitoring and evaluation systems and capacities. They also discussed other important aspects of the partnership, such as its operational principles, budgetary and administrative arrangements, and its governance structure. The workshop concluded with an agreement amongst the participants on key steps and actions they will be taking in collaboration with the co-hosts of the workshop - the World Bank’s Independent Evaluation Group (IEG) and the Independent Evaluation Office (IEO) of United Nations Development Programme (UNDP)  - and other partners towards launching the partnership later this year.  Watch and hear from Wilson Braganca, the Director General of the Ministry of Planning and Finances from Sao Tome to learn more about the global demand for evaluation capacity development from countries. Read more about the current challenges in global M&E capacity and the need for joint action. Sign up to receive updates about the growing global partnership to close the gap in monitoring and evaluation capacity worldwide.

Comoros CLR Review FY14-19

PDF file
This review of the Comoros Completion and Learning Review (CLR) of the World Bank Group (WBG) Country Partnership Strategy (CPS) covers the CPS period, FY14-FY19, and the Performance and Learning Review (PLR) of December 2018. This is the first CPS for Comoros following a series of Interim Strategy Notes (ISNs), the latest of which was prepared in 2010. The WBG programs under the ISNs were Show MoreThis review of the Comoros Completion and Learning Review (CLR) of the World Bank Group (WBG) Country Partnership Strategy (CPS) covers the CPS period, FY14-FY19, and the Performance and Learning Review (PLR) of December 2018. This is the first CPS for Comoros following a series of Interim Strategy Notes (ISNs), the latest of which was prepared in 2010. The WBG programs under the ISNs were limited in scope reflecting the high level of political instability, serious governance issues and related low IDA allocations. The CLR highlighted several lessons about a need to ensure a streamlined project design and flexibility in implementation; value of increased WBG presence on the ground; importance of donor coordination; and a need for greater realism and selectivity in the program. IEG particularly agrees that there is need for greater realism and selectivity in the program, throughout the program, beyond the governance area on which the lesson in the CLR focuses. Being excessively ambitious with respect to institutional targets in a fragile environment increases the risk of program underperformance. IEG adds the following lesson: The decision on a large program expansion at the PLR stage requires a detailed discussion and careful justification in the PLR document because it poses a longer-term implementation risk.

Croatia: Revenue Administration Modernization Project (PPAR)

PDF file
The development objective of the Croatia Revenue Administration Modernization Project was to achieve further improvements in tax efficiency, taxpayer services, and tax compliance through capacity building and systems improvement in the Croatia Tax Administration (CTA). For purposes of this review, three sub-objectives are assessed: (i) improvements in efficiency; (ii) improvements in taxpayer Show MoreThe development objective of the Croatia Revenue Administration Modernization Project was to achieve further improvements in tax efficiency, taxpayer services, and tax compliance through capacity building and systems improvement in the Croatia Tax Administration (CTA). For purposes of this review, three sub-objectives are assessed: (i) improvements in efficiency; (ii) improvements in taxpayer services; and (iii) improvements in tax compliance. Ratings for the Revenue Administration Modernization Project are as follows: Outcome was moderately unsatisfactory, Risk to development outcome was negligible, Bank performance was moderately unsatisfactory, and Borrower performance was moderately unsatisfactory. This assessment offers the following lessons: (i) Poor quality at entry and lack of readiness for implementation contributed to significant implementation delays and limited results. (ii) Given that the main driver of the tax administration reforms was Croatia’s bid for membership of the EU, the project could have better secured the government’s commitment to reforms up front. (iii) In projects aiming to improve tax revenue administration, the right balance must be struck between institutional reform and hardware needs (buildings and information and communications technologies). (iv) High TTL turnover could be mitigated by ensuring adequate capacity in the field with the presence of competent local staff.