The need for self-doubt

I, and many sages wiser than myself, have frequently remarked upon the lack of an adequate feedback mechanism in the aid industry. Randomized Control Trials may be the new M&E gold standard but they suck compared to the rapid feedback of no customers that a businessman with a duff idea experiences.

If the system in which you work has little capacity for self-correction, then, to my mind, it makes it all the more important that we contain such a capacity within ourselves. Which is why I totally agree with Dialectician Terence’s critique of the dangers of over-confidence and self assurance within the development sphere. Whether the road concerned is going to hell, nowhere, or somewhere in between, at best hopes will be raised and money wasted; at worst poor peoples’ livelihoods opportunities could be significantly knocked back.

And yet the conservation and development world holds plenty of attractions for the messianic side that lurks within many of us. Moreover, the need to prise money out of sceptical donors readily pushes us into over-emphasising the potential benefits and pooh-poohing the risks. It can be a short step from there to starting to believe our own guff. Whether it is the Millennium Villages Project or the One Laptop Per Child initiative, we can all too easily find ourselves on a runaway train of propaganda and self-justification.

I’ve had personal experience of working with such people. When they are on the money there is no better ally to be had. But when they lose the plot it can be incredibly difficult to persuade them of the merits of an alternative path. Constructive criticism is dismissed as base politicking, and a multi-stakeholder partnership can, in their eyes, be suddenly transformed into a conspiracy to block progress. When they perceive the project is threatened, then everything becomes about defending the project.

We all have our blind spots, and over-confidence is a dangerous trait in many walks of life. But the project success rate in conservation and development is not good. Even if we have good reason to think our proposal is the best thing since sliced bread, we should be wary, for many before us, equally able, have failed. Thus the capacity for self-doubt is, I think, one of the most important assets you can have if you want to work in conservation or development.

Advertisement

Accounting for failure

Project failure is far too common in conservation and development for anyone’s comfort. Many agencies and practitioners regrettably seek to hide their poor records behind euphemism and by redefining success radically downwards after that fact. Last year an aid bloggers forum considered the question of admitting failure although the consensus was not very positive (see my two contributions: here and here).

Yesterday I blogged about an alternative solution: pushing aid projects to obtain insurance against failure. It’s a nice idea, but probably quite a few years away at best from wide-scale implementation. But, it occurred to me, there is an intermediate solution which would take very little change to implement. Put simply donors would account rigorously for their projects’ success rate. It would work as follows.

Already most donors demand clear statements of project aims and expected outcomes before committing funds. Good donors will also ask for a risk assessment. All we need to do is quantify those risks. Sure putting a number on the likelihood that you will get the necessary buy in from local government officials is an exercise in extreme subjectivity, but we can live with that. Multiply all your risk percentages together and you should get an indication of the likelihood of project success. (I bet often it will be substantially lower than the project’s proponents would like, but if they try to massage it up they’ll get caught out later …)

Then, come evaluation time, the reviewers should explicitly assess what proportion of the original aims and outcomes have been attained, and what risk factors in actual fact came into play to the detriment of project impact. This assessment would have to be extremely robust and refer only to the original estimates of impact, so as not to allow project managers to redefine success downwards mid-project. (A subsidiary assessment could consider revised aims that were formally set out and agreed.)

Project proponents and implementers who consistently underestimate risks will be shown up (although, obviously the feedback loop will take a few years to generate much in the way of information). Imagine if donors pooled all this information so that they could look up organisations records. Imagine further if such estimates were linked to specific key people who worked upon the projects, and you had to justify both your risk assessment accuracy rate and actual project success rate in your next job interview. (Sensible employers would be tolerant of those who have only worked on a few projects and just got unlucky, but could talk intelligently about what went wrong, the lessons they learned and how they would put right such situations in future.)

As well as improving accountability and honesty about the very real risks involved in most conservation and developing projects, as the data built up, donors could also use it for their own internal evaluations. How successful were their projects? Which types of risk factors proved to be the most dangerous? Which types of risk factor were consistently under-estimated and which might have been over-estimated? Donors wanting specifically to target riskier projects with some or all of their money should not be discouraged; we all know that the pay-off from such initiatives can be that much bigger than me-too carbon copies of established models. This way the risk would simply be more explicitly acknowledged.

The total guesswork inherent in the original risk estimation would limit the data’s utility in evaluating individual performance, but for donors and BINGOs, when aggregated across the organisation, these errors would start to average out, and analysis of later results would help agencies to refine their estimates. E.g. typical political risk factors could be classified according to severity, with information for project proponents on actual failure rates in previous projects to help them gauge the likely risk in the new project they are proposing.

I can see plenty of resistance from many in the aid industry to such crude quantifications, but the move to increase transparency in the sector is gathering momentum. Perhaps it is the sort of thing that could be considered in the next iteration of IATI?

Time to call in the lawyers?

Mike Jennings suggests homeless Haitians should be able to sue international NGOs for leaving them in temporary tented camps for so long, and not providing more permanent housing. You do not have to know anything about Haiti (I don’t!) to see how this would be a spectacularly bad idea.

We can start off with a roll call of those countries with strict NGO regulations: North Korea, Burma, Sudan, Ethiopia … The list goes on but you get the idea.

Dr Jennings is a lecturer at the School of Oriental and African Studies (SOAS) at the University of London, so may be familiar with the results of the growing litigation culture in the UK with respect to public services. It’s affected the health sector, but I know more people in education. There it has resulted in an extremely risk averse approach; pupils are taken on far less school trips because schools and teachers are scared of getting sued. Just the paperwork involved for the simplest trips can put teachers off organising them. The outcome is a poorer, narrower and less engaging education for our children.

Now let’s compare with humanitarian aid. At least in educating our children teachers and schools mostly know what they’re doing. Despite the odd ideological battle, we more or less know how to educate a large proportion of our children to become welcome members of society. In contrast, the long list of disappointing outcomes from various humanitarian aid missions and projects shows just how little we understand what makes for a good project.*

If NGOs were to be made legally liable for delivering outcomes to some minimum standard (as determined by whom??) then they would retreat into the least risky kind of projects. Instead of a tented camp for destitute refugees they might provide scholarships to study in the US and Europe – not much that can go wrong there, but only a tiny number of people (probably already middle class to have the necessary education to benefit from tertiary study overseas) and substantial cost per person supported.

Mike Jennings appears to acknowledge this problem when he says:

By their nature, efforts to alleviate poverty, suffering and vulnerability in some of the most economically, socially and politically challenging parts of the world are risky. But risky for whom? Risky for the organisations who may face financial difficulties or a loss of prestige? Or risky for the communities in which the intervention has taken place?

I’ve never worked in emergency humanitarian situations such as post-earthquake Haiti, but I would imagine for most beneficiaries their initial risk was very low, as they had lost a large part of what they had before, and were likely dependent on outside assistance for the most basic needs such as food and clean drinking water. In general, in the aid industry, we strive to ensure that aid dependency is only short term, but it can be awfully difficult for both sides to extricate themselves from that rut once they are in it. Compared to where they started I cannot see what the big downsides are for people so desperate. This is not to say that we should not aim to give them the very best assistance we can for a given input of resources, but it seems perverse to me to paint such people as ‘victims’ of the international aid industry.

My experience, however, is in non-emergency work, and actually I think poor communities can be quite good at risk management. Although, by nature, conservative (a sweeping generalization, I know) communities welcome most proposed projects with open arms even where they do not fully understand the project design. However, they do know they’ll get two things: investment into their community of some form or another, and participants will be paid per diems for their time. As project implementers we hate this requirement as it smacks of aid dependency; we shouldn’t have to pay people to be permitted to help them. But that’s the way it is, and that way the community can be assured as to some benefit even if the project peters out in a few years without ever coming close to achieving what it set out to do. The development professionals will move away to the next big thing / step up on their career, and the community got some pocket money. Plus ça change …

All of which reinforces the need for strong accountability and transparency in aid work – and not just to the donors – in which all aid actors perform poorly (don’t just blame the NGOs), that lay behind Mike Jennings’s proposal. Just please leave the lawyers out of it!

* Actually I think quite often we do know how to do a successful project, but either the donor wants to reduce the costs and cut the time available, and/or the recipient country politics constrain the list of options, but that still leaves a long list of things we still really do not know how to do, such as providing new permanent accommodation in a country with a completely dysfunctional land registry.