After attending the UNDP National Evaluation Capacities and the American Evaluation Association Conferences in 2019, I wrote a blog Are Evaluators Ready to Answer Question: Who Benefits? The answer was that increasingly evaluation practice was grappling with ‘Who Benefits’ through increasingly engaging with issues such as inequality, inequity, the differing impacts of climate change, and the role of youth. This year I attended the virtual American Evaluation Association (AEA) conference interested in how the debates on assessing benefits had developed.
The theme of the AEA conference was ‘How Will You Shine Your Light’, which relates to the idea that evaluators shine their light for the purpose of “improving conditions for others to shine theirs”. Throughout, the conference focused on equity, the fair distribution of benefits. The presidential track of the conference reflected on equity through discussions on restorative practice, racism in qualitative practice, disability in evaluation, community-based foresight practices, climate change and presenting a framework for assessing community harmony. Outside of the presidential strand, presentations also focused on gender equity, equity-based policy advocacy and equity and inclusion.
The emergence of equity as a major theme in evaluation practice reflects trends where evaluation gatherings embrace a diversity of perspectives, including indigenous, youth and social justice perspectives. Many of the presentations at the conference reflected critically on approaches to engage marginalized groups and called for more context specific and culturally appropriate methods. Some of the interesting resources that I would reference again as I develop evaluations were related to youth, the rubric developed to measure love, equity in data science and foresight techniques.
Doing No Harm in Evaluation
Harm is not a word that evaluators have engaged with deeply very often. The presenters of this Think Tank initially came together to reflect on harm in evaluation practice when organizations that were meant to protect rights but caused harm were exposed. Over a period of two years we have further understood that evaluation can cause harm through its: handling of allegations of sexual exploitation; inaccurate findings; privileging of voices; myopically focusing on donor values; careless storing of data; cultural insensitivity; and not evaluating harm. This think tank introduces these issues and seeks to identify where our principles and standards need updating to better reflect the potential for harm. The Think Tank will engage participants in a participatory reflection process aimed to develop adaptations for where our principles are found wanting.
My understanding of the ‘Who Benefits’ question was especially deepened in facilitating a Think Tank entitled Doing No Harm in Evaluation that drew on a two-year action learning process. Discussing harm is the ‘shadow’ of the equity and benefits discussions, as one of our discussants highlighted. Evaluators and project managers who have the best of intentions and seek to diversify who, what and how benefits accrue can do harm – think of abuse in humanitarian settings.
In debating harm, the participants went beyond concerns with ethics panels and informed consent. They questioned how harm could be caused by funding and commissioning mechanisms, a tendency to drop into contexts with limited preparation, and the answering of questions that were defined far from intended beneficiaries. Emerging research that was presented within the Think Tank made the argument that in evaluation there is inherent reductionism and carcerality, which requires changing listening practices, reciprocity with knowledge, and undertaking truth-telling on harm.
The online chat during the think tank produced just less than 300 comments in 45 minutes. In Interpreting these afterward the following five themes emerged:
- Trade-Offs – In identifying who benefits, harm is embedded. Consequently, trade-offs between benefit and harm should be acknowledged in the evaluation processes, for example, in the (i) methods deployed; (ii) scope of the evaluation; (iii) engagement with commissioners; (iv) management process.
- Reflexivity – Evaluators hold a position of privilege in defining benefit and should give attention and attend to personal growth in their own judgements, beliefs and practices as benefits can be defined in diverse ways.
- Trauma informed and restorative practices – Identifying who benefits may also bring an evaluation into contact with past traumas. Trauma-informed and restorative practices arose as approaches to help account and tell the truth on the harms of past practice while providing a framework to help manage trade-offs in current evaluation practice.
- Inclusivity – To understand who benefits requires improvements in h who evaluation practice includes, through listening skills and incorporating feedback loops. Creating space for listening will require responding to new and unexpected information on diverse perspectives.
- Transformative lens – Working with a transformative evaluation perspective helps to identify different benefits and harms by reinforcing attention to voice and liberation in discussions. Transformation provides potentially a useful ethical principle in reframing practice to better respond to harm.
These themes and the discussions presented here reflect a small part of exchanges on equity that evaluators engaged in during the conference. Terms such as trade-offs, reciprocity, restorative, reflexivity, transformative are increasingly eminent in evaluation practice and represent a deepening discussion amongst evaluators to engage with ‘Who Benefits?’