Looking beyond the price when assessing the cost of an evaluation

In the last couple of posts in this series on value-for-money in evaluation I looked at value and how to measure it. Today, I want to turn to the question of money and the relationship between price and cost.

 In Value-for-Money in the Evaluation Business I wrote about the perceived or actual "price" of a negative evaluation: a bad scorecard, an embarrassing assessment, or worse - fear of losing a project, a job, or a funder.

Today, I want to unpack the question of cost, both direct and indirect.

Our attention often gets absorbed by direct evaluation costs -€“ what is the budget, what have we spent it on, how much do consultants and fieldwork cost -€“ which are important considerations, especially when there is so much data and information available that needs to be considered in ever more complex projects, programs, or policies. At the same time, efficiencies can be achieved with new technology that allows for the analysis of data and text in ways that surpass more time-consuming methods that potentially put evaluations at risk of inconsistencies. Access to information ahead of time can also reduce the amount of fieldwork (where a lot of the cost sits) required.  

But in discussing value-for-money, we also need to look at indirect costs - sometimes monetary, often not - that can enhance or diminish an evaluation.

Transaction costs. All of us know how "expensive" meetings are: they take a lot of time, distract us or take time away from other things, and depending on how they are run, they can feel like an utter waste of time. For evaluators meetings can be particularly tricky as the people we'€™re talking to may be reluctant to share information during the evaluation interviews or may fear negative results when discussing draft findings.

My own approach to turning this handicap into an advantage and eventually a value preposition?

Be extremely well prepared: This requires reading all there is to read and understand prior to meeting stakeholders. It has enabled me to engage with people in a deeper conversation about what matters to them most, their work, their project, the results they want to achieve, and what stopped them from doing so. When they see that the evaluation is addressing their problems and recommending ways to fix them, they appreciate its value.

Increase the quality of the engagement:  How we engage with stakeholders can determine the cost of our evaluation and its value-added. For instance, meetings that tackle problematic findings in a frank and constructive way enrich the evaluation process, while engagements that avoid key problems or deal with them in confrontational ways are not productive and could increase the cost.

Cost of quality assurance. Quality is paramount to evaluation! Poor quality will come at the cost of reputational risk and reduce the effectiveness of evaluation. In addition, there is a real cost to quality assurance - the time needed to review, discuss and revise reports. What can help minimize these costs, without making the wrong trade-offs about quality, is having a clear and agreed understanding of standards and expectations, and using modern techniques that help triangulate evidence from different sources to crystallize well-sourced, clear messages. Getting it right is hard, but it is also indispensable if we are to increase the value of evaluations while reducing the cost of their production, both in terms of time spent and in credibility vis-a-vis stakeholders.

Cost of discussing draft findings. Related to the cost of quality assurance, is the cost of engagement with stakeholders on the draft final report. There is the simple time factor where everyone'€™s "time is money"€. In addition,

Cost of sharing lessons. Very often, the cost of an evaluation is estimated to include only the work up to when the report is completed and published. Encapsulating evidence and knowledge in a document without sharing it diminishes its value. Getting more mileage -€“ more value - out of evaluations requires additional investments in disseminating their findings. In the past, that might have meant simply sending copies of the evaluation to interested parties. Today, we have a panoply of communications tools that help us to not only share the information, but also to tailor and package it to specific audiences who otherwise would not read an entire, often lengthy and detailed, evaluation report.

In short, there are costs we do not count, but that can make the difference between an evaluation that is costly and one that brings added value.