Project failure is far too common in conservation and development for anyone’s comfort. Many agencies and practitioners regrettably seek to hide their poor records behind euphemism and by redefining success radically downwards after that fact. Last year an aid bloggers forum considered the question of admitting failure although the consensus was not very positive (see my two contributions: here and here).
Yesterday I blogged about an alternative solution: pushing aid projects to obtain insurance against failure. It’s a nice idea, but probably quite a few years away at best from wide-scale implementation. But, it occurred to me, there is an intermediate solution which would take very little change to implement. Put simply donors would account rigorously for their projects’ success rate. It would work as follows.
Already most donors demand clear statements of project aims and expected outcomes before committing funds. Good donors will also ask for a risk assessment. All we need to do is quantify those risks. Sure putting a number on the likelihood that you will get the necessary buy in from local government officials is an exercise in extreme subjectivity, but we can live with that. Multiply all your risk percentages together and you should get an indication of the likelihood of project success. (I bet often it will be substantially lower than the project’s proponents would like, but if they try to massage it up they’ll get caught out later …)
Then, come evaluation time, the reviewers should explicitly assess what proportion of the original aims and outcomes have been attained, and what risk factors in actual fact came into play to the detriment of project impact. This assessment would have to be extremely robust and refer only to the original estimates of impact, so as not to allow project managers to redefine success downwards mid-project. (A subsidiary assessment could consider revised aims that were formally set out and agreed.)
Project proponents and implementers who consistently underestimate risks will be shown up (although, obviously the feedback loop will take a few years to generate much in the way of information). Imagine if donors pooled all this information so that they could look up organisations records. Imagine further if such estimates were linked to specific key people who worked upon the projects, and you had to justify both your risk assessment accuracy rate and actual project success rate in your next job interview. (Sensible employers would be tolerant of those who have only worked on a few projects and just got unlucky, but could talk intelligently about what went wrong, the lessons they learned and how they would put right such situations in future.)
As well as improving accountability and honesty about the very real risks involved in most conservation and developing projects, as the data built up, donors could also use it for their own internal evaluations. How successful were their projects? Which types of risk factors proved to be the most dangerous? Which types of risk factor were consistently under-estimated and which might have been over-estimated? Donors wanting specifically to target riskier projects with some or all of their money should not be discouraged; we all know that the pay-off from such initiatives can be that much bigger than me-too carbon copies of established models. This way the risk would simply be more explicitly acknowledged.
The total guesswork inherent in the original risk estimation would limit the data’s utility in evaluating individual performance, but for donors and BINGOs, when aggregated across the organisation, these errors would start to average out, and analysis of later results would help agencies to refine their estimates. E.g. typical political risk factors could be classified according to severity, with information for project proponents on actual failure rates in previous projects to help them gauge the likely risk in the new project they are proposing.
I can see plenty of resistance from many in the aid industry to such crude quantifications, but the move to increase transparency in the sector is gathering momentum. Perhaps it is the sort of thing that could be considered in the next iteration of IATI?