Accounting for failure

Project failure is far too common in conservation and development for anyone’s comfort. Many agencies and practitioners regrettably seek to hide their poor records behind euphemism and by redefining success radically downwards after that fact. Last year an aid bloggers forum considered the question of admitting failure although the consensus was not very positive (see my two contributions: here and here).

Yesterday I blogged about an alternative solution: pushing aid projects to obtain insurance against failure. It’s a nice idea, but probably quite a few years away at best from wide-scale implementation. But, it occurred to me, there is an intermediate solution which would take very little change to implement. Put simply donors would account rigorously for their projects’ success rate. It would work as follows.

Already most donors demand clear statements of project aims and expected outcomes before committing funds. Good donors will also ask for a risk assessment. All we need to do is quantify those risks. Sure putting a number on the likelihood that you will get the necessary buy in from local government officials is an exercise in extreme subjectivity, but we can live with that. Multiply all your risk percentages together and you should get an indication of the likelihood of project success. (I bet often it will be substantially lower than the project’s proponents would like, but if they try to massage it up they’ll get caught out later …)

Then, come evaluation time, the reviewers should explicitly assess what proportion of the original aims and outcomes have been attained, and what risk factors in actual fact came into play to the detriment of project impact. This assessment would have to be extremely robust and refer only to the original estimates of impact, so as not to allow project managers to redefine success downwards mid-project. (A subsidiary assessment could consider revised aims that were formally set out and agreed.)

Project proponents and implementers who consistently underestimate risks will be shown up (although, obviously the feedback loop will take a few years to generate much in the way of information). Imagine if donors pooled all this information so that they could look up organisations records. Imagine further if such estimates were linked to specific key people who worked upon the projects, and you had to justify both your risk assessment accuracy rate and actual project success rate in your next job interview. (Sensible employers would be tolerant of those who have only worked on a few projects and just got unlucky, but could talk intelligently about what went wrong, the lessons they learned and how they would put right such situations in future.)

As well as improving accountability and honesty about the very real risks involved in most conservation and developing projects, as the data built up, donors could also use it for their own internal evaluations. How successful were their projects? Which types of risk factors proved to be the most dangerous? Which types of risk factor were consistently under-estimated and which might have been over-estimated? Donors wanting specifically to target riskier projects with some or all of their money should not be discouraged; we all know that the pay-off from such initiatives can be that much bigger than me-too carbon copies of established models. This way the risk would simply be more explicitly acknowledged.

The total guesswork inherent in the original risk estimation would limit the data’s utility in evaluating individual performance, but for donors and BINGOs, when aggregated across the organisation, these errors would start to average out, and analysis of later results would help agencies to refine their estimates. E.g. typical political risk factors could be classified according to severity, with information for project proponents on actual failure rates in previous projects to help them gauge the likely risk in the new project they are proposing.

I can see plenty of resistance from many in the aid industry to such crude quantifications, but the move to increase transparency in the sector is gathering momentum. Perhaps it is the sort of thing that could be considered in the next iteration of IATI?

Advertisement

Insuring against failure

With Rio+20 about to open I have to give a shout out to the idea of iREDD advanced by Corey Bradshaw et al. They propose to address some of the major challenges of REDD by requiring sellers of REDD credits to purchase insurance (that oh-so-cool i on the front of iREDD). One very good point in its favour is that this should massively increase the buyers’ confidence, although, it should be noted that voluntary carbon market standards such as VCS already require project developers to retain substantial buffers of carbon offsets for just such an eventuality. However, I think financial insurance is a stronger option, since one catastrophe – e.g. a massive forest fire – could wipe out a project’s entire carbon achievements with no recourse. Insurance provided by a highly capitalised third party delivers much better cover for such events.

Thus insurance appears a very good way of dealing with the problem of permanence. It should also be fairly easy; there are pretty good data available on wild fires, whilst forest owners who themselves cleared the forest (or allowed someone else to do so) would be guilty of insurance fraud. (Following the logic of my post yesterday, insurers would probably also have to consider political risk, in which the forest owner suddenly finds they are no longer the forest owner.)

But in tackling the other challenges in REDD in demonstrably delivering real net additional reductions in carbon emissions (see here for my previous observations) iREDD seems mostly just to come down to the idea of insuring against the risk that a proposed action does not result in the desired outcome. Here I see a much greater challenge for insurers. How should they quantify such risks? Every project will be unique in some way or another. I foresee great difficulties in developing standard metrics by which these things can be assessed, and where such guidelines could be determined I fear they would tend to favour cookie-cutter style projects and incentivise against innovation.

On the other hand, if these hurdles could be overcome, this seems to me like an idea that has great potential way beyond the REDD arena. How about if donors required every aid project except for the riskiest (which would therefore obviously stand out) to obtain such insurance? That would force project developers to confront major operational risks in a much more explicit manner than at present. Furthermore it could open up innovative approaches to project funding (donors would expect to get some of their money back when projects fail), and could allow private sector operators taking a greater responsibility for the entire project cycle and assumption of risk. This would be in contrast to the current model for involving the private sector in which big consultancies make fat profits for running flawed projects designed by a donor who should have but didn’t know any better.

Owen Barder recently blogged about attempts to extend the still experimental idea of Social Impact Bonds into the international development space*. These also invite private sector players to assume some of the risk of delivery. Insuring projects against failure could be another option to add into the mix.

In practice I can see that it is going to take quite a while for these various different instruments to be put into practice, during which time we can hope – indeed reasonably expect! – that significant numbers of poor people and poor countries will have developed to the point where they are rather less poor and less in need of development assistance. But that could also work in these ideas’ favour: the least developed countries have the least capacity to engage in these kind of risk-sharing models involving the private sector.

So all in all I think it is great to see innovative thinking around conservation and development finance, and I hope that at least some can come to serious fruition. And I look forward to one day filling out an insurance registration for a project I have helped design!

* The various comments echo my points above about some of the practical challenges that would need to be overcome to introduce these as financial instruments worthy of the name.

When admitting failure isn’t enough

There have been some great posts on the second aid blog forum on admitting failure.  Many bloggers picked up, as I like to think I did, on the fact that admitting failure is just one aspect of lesson learning (another tautological piece of yucky aid jargon), that we all ought to be doing as a matter of course. David Week called attention to this better than anyone, demolishing admitting failure as just another management fad. (I find it hard to disagree with him, but reckon admitting failure has so much more humility than your average management fad, that I’m prepared to give it the time of day.)

In particular, David examined the failure reports by Engineers Without Borders, the poster child for admitting failure in aid projects. In doing so he highlighted the limitation that I had suggested, that failures identified and admitted were unlikely to be central to an organisation’s work, but focus on relatively peripheral elements. David dissected an entry in EWB’s 2009 failure report. To paraphrase: he showed that while the EWB volunteer had successfully identified that things had gone wrong, and that the project would not be sustainable, she had failed to identify that the real problem lay in the fact that the whole project design was fundamentally unsustainable in the first place.* Or as I suggested in the comments: EWB is exposed as a glorified volunteer monger. Maybe one of the best volunteer monger’s out there, but a volunteer monger nonetheless.

EWB’s Erin Antcliffe responded in the comments and an excellent little debate developed, spreading to Twitter.** Now don’t get me wrong; I have followed several EWB volunteer blogs over time. I love their questioning approach and courage to face up to failure. Without ever having been near one of their projects, I nonetheless imagine they might just be the best volunteer monger in the world. And if their initiative to get aid and development organisations to similarly face up publicly to their failures catches on, then, regardless of David Week’s and others’ reservations, I think they’ll have done the world a big favour.

But, will their project design process have changed? Will they have learned the most important lesson from their failures? I can see this how might be difficult, because it appears fundamental to how the organisation works.

Often times in this blog I have contrasted how the topsy-turvy world of aid differs from that of business. (As have many others wiser than me!) This appears to be another such example. If you came up with a great new business idea, you could give it a good go, but if, whatever your original genius, it failed to deliver you would find out pretty quickly and the company would collapse. Alas the absence of good feedback loops in aid means that as long as you can convince the donors to keep on donating (and the volunteers signing up), you can go on indefinitely regardless of what you actually deliver.

It’s hard enough to admit failure in the first place. It’s even harder to admit that you might actually be the problem. And what matter most is what you do after you’ve admitted failure.

* This raises an important point. It is not enough simply to admit failure. One then needs to correctly diagnose the cause. This is not always easy!

** You can now follow me on there too: @bottmupthinking. Don’t count on too regular tweeting.

With fails like these who needs success?

This is a contribution to the second aid blog forum on admitting failure in aid projects. Several contributors have already pointed out the challenges of admitting failure in the first place, and I don’t want to pooh pooh their very real concerns. But I also think this is an idea whose time might be just around the corner. Once the ground has been broken by a few brave NGOs and supportive donors there could suddenly be a big rush, all in the name of marketing. It could come de rigeur to admit to at least a few failures.

All very wonderful, but I worry that a lot of it’s going to be rather superficial. As Marc Bellemare points out, in admitting failure, on one level all we’re really doing is showing that we are sufficiently self-critical to do so, and thus earn more kudos for self-criticism than we lose for the odd failure. This reminds me of those long application forms for graduate level jobs and questions such as ‘Name three weaknesses you have.’ No graduate worth their salt is going to confess that they don’t get on well with other people, but instead will say things like ‘can be a bit impulsive from time to time’ which can be almost turned into a positive.

Similarly, no NGO is going to admit that big chunks of their work is a complete failure. Instead, like the example given by Tom Murphy, they’re going to contrast their failure with a success (thus emphasising the value of the success story) and/or pick on relatively minor elements of their work. Of course this is just what you’d expect from human nature, and sensible management. Nobody wants to do a Ratner!

On the flip side of the coin, I would expect this to be a reasonable reflection of reality in well run organisations. Such organisations should be capable of spotting when they’re heading for failure on a major programme and devote the management time to turning things around. That’s what adaptive management, the mark of a good development project just as it is the mark of good business management, is all about. Thus the failures ought to be peripheral; where senior staff just took too long to become apprised of the trap into which they were about to fall. A good organisation should also be able extricate themselves from any such traps. Indeed admitting failure then just becomes another element of the lesson learning process that an effective organisation should be going through internally anyway. (Thus arguably admitting failure is simply exposing that process to the outside world.)

The trouble is that a lot of what goes on in development does not appear to me to be that well run. (On average I’d say the quality of management in NGOs is probably better than other parts of the aid world, but it’s not a hard and fast rule.) Will these less well run organisations, programmes and projects have the courage to admit they made much bigger failures? I fear not. Just ask the leading advocates of the Millennium Villages Project!

That all said, I can see a big potential win here if admitting failure really takes off. Because for all the lack of self-criticism in the aid world, I think it is worse in developing country governments. So if self-criticism became that much more mainstream, then there is a chance that it might percolate across the institutional boundaries. I’m not overly optimistic on this point, but it is certainly worth the attempt.

To conclude, I would have failed myself if readers of this post came away with a negative impression of the growing fad for publicly admitting failure in development projects. I think it is an excellent innovation, and I hope it catches on. A little more humility from the big aid players would be no bad thing any way. Just expect a certain degree of superficiality and turning negatives into positives along the way, because that’s just human nature.

The use of euphemism and excuses for failure

In amidst all the usual jargon, international development includes some truly excellent / excruciating euphemisms. ‘Rent seeking behaviour’ aka corruption (or gangsterism, bullying, cheating and stealing) has to be my favourite, whilst ‘Development Partner’ aka donor (he who pays the piper, not exactly an equal partner) is one of the most ridiculous. This reluctance to call a spade a spade is very understandable in the context of international diplomacy – indeed we could probably not do without it – but is less helpful in a results-focused business which is what the aid industry these days claims it is.

Although the two examples above do not particularly relate to project performance, much of this euphemistic language arises in attempts by donors and implementing agencies to explain the failure of their last development project. The preference is always to find some technical or at least technically sounding (hence the need for euphemism) excuse as to why a certain project failed. Although Ben Ramalingan was talking much more generally than about the failure of a single project when he criticised the Results-Based Management framework that I discussed in my previous post, the wrong management framework is another good candidate. Anything is better than criticising the recipient country managers; just because they may be incapable of organising a booze-up in a brewery it isn’t their fault. If anything they just need their capacity building …

So we build their capacity. We send them on a few training courses. Yeah! Now they know how to use a logframe everything will go swimmingly … Rarely does the aid industry really attempt to get to grips with the real capacity constraints; poor management culture and incentives in the civil service. (I’m sure there are more.) Donors know reforming the civil service is hard enough in their own countries, and World Bank supported efforts in developing countries grind along at a snail’s pace achieving only peripheral successes, e.g. performance appraisals without performance related pay or promotion.

I think this partly explains the constant search for new ideas and potential silver bullets in development, even though we actually have quite good ideas already of quite a lot of things that work … when managed properly. When the great new hope comes along – e.g. REDD in conservation – the taps open once again, and all the same mistakes are made over again. “This time it’ll be different”, donors – sorry, development partners! – tell themselves, because, well, hope springs eternal.

This is not a call to end development aid; not all aid projects fail and far from all developing country managers are incompetent. But I do think the industry is going to have to get more honest with itself. We need to set more modest targets for aid projects and stop using implementation channels that are known not to work. Then we need to fess up when things don’t work, and end the self-delusion as to why they didn’t work. International diplomacy can work with appropriate technology human-wielded latrine construction tools, but successful development just needs a few spades.

It’s the quality of management, stupid

This is a familiar refrain of this blog, but I was reminded of it by Ben Ramalingan’s dissection of the failures of Results-Based Management in (UN funded) aid projects. He makes some good and depressing points about the inflexibility in approach and other limitations introduced by the RBM framework. But here’s the thing: these sorts of challenges (‘wicked problems’) are confronted on a regular basis by businesses all around the globe. Many business problems cannot be solved by some kind of linear engineering algorithm, but require flexible, creative and iterative solutions. That this is not easy is reflected in the large number of businesses that fail every year, but plenty succeed.

How many development interventions succeed? I have no idea, since admitting failure is one of the things the aid industry finds hardest, but we can hypothesise that the success rate is lower than that of the private sector. If we accept this hypothesis for a moment, the obvious question is to ask why the lower success rate? I would venture to suggest that the answer lies in the quality of management.

Before someone points out the obvious, I should add here that I’m talking about relatively simple project level interventions, e.g. improve access to clean water in a defined area, rather than the really wicked problems of conservation and development such as how to eliminate poverty or stop deforestation. Aid projects to deliver such services seem reasonably analogous to businesses and thus the comparison is fair.

Ramalingan is right to criticise restrictive management paradigms brought to bear in development projects, but I think he’s nonetheless missing the point. A good manager will always adapt and refine plans to reflect the reality with which they are confronted, and good management systems will allow good managers appropriate leeway. This is basic good management principles and I don’t see why it shouldn’t be achievable with RBM (though I’d recommend a good dose of KISI too). Inadequate or long delayed feedback on progress, a much lamented challenge in development, also shouldn’t be such a problem at the project level (it is much harder at the planning / donor level), as a good manager should always have a pretty good idea as to whether progress is being made, e.g. determining whether you’ve been successful in supplying clean drinking water to poor people isn’t rocket science!

So how does good business management differ from that used in a typical development project? Here are some suggestions:

• No fixed delineation between planners and implementers (see my earlier post), instead there will be regular communication between senior and junior managers.
• This typically leads to more flexibility to adapt when originally adopted strategies don’t work so well.
• Clear lines of responsibility: business is a much less collaborative environment than development. Managers have reasonable authority to act within the realms of their responsibility, and can quickly refer upwards if needed. Contrast with bilateral donor projects riven by the gaping chasm between government staff on one side and expat technical advisers on the other.
• Successful businesses have management teams that stay together for long periods of time, and thus have learned how to keep a steady hand on the tiller. Businesses know that if their key talent in a certain area leaves their business interests there will likely suffer or even collapse entirely, so great care is taken to ensure continuity when staff do leave. Contrast with rotation of expat TAs on development projects, who often only stay for 2-3 years before moving on.

So sure, having a good planning / management framework can help, but it is the soft skills that matter so much more, and that’s why most businesses, especially those operating in the knowledge / service economies, place such a high importance on good HR management. When will aid / development / conservation learn this lesson?

Don’t forget how it was before

Continuing the spirit of new starts in the new year, I was intrigued by J’s post considering to what extent aid efforts have succeeded in Haiti since last year’s devastating earthquake. His post mostly talks about the huge scale of the disaster and complexity of helping its surviving victims, and how these challenges explain the lack of success or even “failure” of interventions thus far. But how did it all start out?

Obviously all development and humanitarian relief interventions have target end goals they are seeking to meet, and when they fail to achieve the desired end point there is always a degree of failure to be considered. However, it should be remembered that most such goals are what the business world calls stretch targets. Indeed it is often necessary to be deliberately over-ambitious in order to secure funding. That is to say the architecture of the aid and development sector sets up many projects to apparently “fail” in the first place.

J, however, also questions whether it is correct to talk in terms of success and failure, and I think he is right to do so. When coming to a new place in the developing world it is always easy to spot the problems; complaints are often not far behind. What these critiques may fail to understand is just how bad the situation was before, and thus the degree of success that has been achieved.

This applies not just to disaster zones, and I was reminded of it over my Xmas break. We went to visit some friends in another developing country (it doesn’t matter which). I knew the country very little, and our friends not much better (they’ve been there just over a year). Of course, before long our conversation turned to the various shortcomings manifest in the government of said country. It wasn’t hard to find things to criticise, and there is certainly much that could be done better. However, our friends also passed on the views of some of their local friends and longer term residents. They were more relaxed about things; yes they agreed with all the criticisms but nonetheless they said the political situation was much improved now upon a few years ago. Things were bad, but they had been worse; the country, still very poor, is on an upward trajectory (although unlikely to become rich any time soon).

I help run a small NGO. Some of our management processes are still necessarily somewhat chaotic; we’re still putting together all the systems necessary to make it run smoothly. But compared to where we were ~5 years ago the organisation is a purring machine of efficient endeavour. It’s hard for new staff to realise that, though; they just see the problems (not enough computers) and don’t appreciate the progress we’ve made (2 laptops between 4 of us when we started).

Of course, one must be careful of this argument. It is not an excuse for every missed project milestone or impact that goes unachieved. The defence that things would have been much worse without the project only goes so far, and is probably over-used. But progress is something  to be valued. When evaluating a project, and deciding whether to commit further funding, donors need to be careful not to throw out too many babies with the bathwater. Here, unfortunately, I detect double standards: NGO projects can be measured against impossibly tough yardsticks, whilst bilateral donor projects continue to pour funds down recipient government maws whilst negligible progress is made towards the targeted outcomes.

That is a subject to explored more fully in a future post. For now, whenever you go somewhere in the developing world, don’t forget to ask how it was before, before you go making judgements about how it is now.