Burying criticism

Chris Blattman thinks that NGOs are atrocious in the way they bury criticism. This was his opinion given in response to a paper showing NGOs were twice as likely to want to work with outside, independent academics who appear to have an a priori view that is supportive of the sector in which they work (microfinance in the study) than they would if the a priori view was critical.

I work for an NGO and largely agree with the Blattman. Indeed this blog has covered NGO unconstructive responses to criticism before (here and here). I also like the study design which reminds me of those experiments where identical CVs were sent out but with different names on top to test latent racist attitudes. But I think there are some important points to make in response.

1. As pointed out in a comment on Blattman’s post, in the study the negative treatment email’s opening sentence of paragraph 2 is pretty uncompromising: “Academic research suggests that microfinance is ineffective.” This blunt approach, which mirrors exactly the positive treatment email, unfortunately does not conform to social norms in how we express disagreement, which is usually hedged in a way to give due acknowledgement of the correspondent’s position. In short the negative email approach does not make them sound like great partners.
2. People in glass houses shouldn’t throw stones. We all struggle to deal with countervailing views. Science is littered with examples of where prominent academics of the time have suppressed publication of a paper that ran counter to their view of the world, only for it later to turn out the author was correct and the bigwigs wrong.
3. NGOs live, work and fundraise in the real world, not some idealised planet populated by homo rationalis. The robust battle of ideas that lies at the heart of academic life may be the anomaly, not NGO responses to criticism. Inaccurate or inappropriate criticism can do a lot of damage to an otherwise successful programme. NGOs may conclude it is better to do their learning behind closed doors, and then change direction if appropriate, than to admit their failures. Like it or not, I suspect many of the most successful NGOs are probably those which more carefully burnish their reputations by controlling the release of bad news about their work.

However, in my view, none of the above constitute sufficient justification for the lack of openness amongst many NGOs in the conservation and development sectors. The time has come for admitting failure and other exercises in honest self-assessment and humility. Though the road may be tough, eventually we will all be stronger for it.

On whose behalf do they speak?

How Matters recently featured a guest post by Scott Fifer on the need to listen to local leaders, especially when they say “Thanks but no thanks.”

As with the paper on which I commented yesterday, Scott was making one of those obvious points which are too often forgotten, and I found myself agreeing with just about every word he wrote. However, I was also a tad concerned that the excellent points he was making all depended on one critical question which was never fully addressed: to what extent is Scott’s local partner, Abel Barrera Hernández, actually a good representative of the people Scott and Abel want to help?

Us expatriate advisers often worry a lot about the extent to which we should speak on behalf of the beneficiaries we are seeking to support, but can suffer serious blind spots when it comes to our local partners. As far as I could tell from Scott’s post Abel has not been elected by the people of Guerrero, Mexico. Even if he is extremely popular, and has helped the people there a lot (Scott implies both), they might not agree with Abel when he argues against the GO Campaign providing funds to improve the local school with a new floor on the basis that this ought to be the government’s job.

“Admittedly, part of me wanted to say to Abel, the hell with the cultural traditions, and to hell with the government! These kids need an education. These kids want and need books and desks and chairs and a floor.  But that part of me shut up (mostly).  My years of grant-making have taught me to know that I don’t know it all. And if a respected leader and human rights champion is telling me my well-intentioned ideas don’t fly with him, then I gotta figure he knows more than I do.”

It seems that Scott and Abel came up with a solution that sounds sensible and will give the practical support to the communities they need, so, without knowing anything more about the situation in Guerrero, I do not wish to imply any criticism of this particular instance; indeed the conclusion they reached sounds praise-worthy. But we do not always get it so right with our partners selection, e.g. in Guinea-Bissau it appears IUCN put rather too much trust in their local partners when they embarked upon a community conservation initiative on the Cubucan peninsular.

Local partners are vital to the success of development projects, but just because they are local and they are your chosen partner, does not necessarily make them good, nor automatically entitle them to speak on behalf of your would-be beneficiaries. Both labels have to be earned, and sometimes you might even find that actually it is the international partners who do a better job of protecting the interests of the beneficiaries.

Standing on your head

The pioneers of community-based natural resources management framed it firmly within the bottom up approach to development paradigm, and yet donors, recipient governments and the big multi-laterals (and some BINGOs) often find it really difficult to shake the top-down mentality.

“[Community-based fire management] training workshops designed to increase the expertise of practitioners should be conducted at the national and sub-national levels and should be followed up with an adequate level of technical support.”

That is the second practical recommendation made in the Executive Summary (not an obscure bullet point buried in an appendix) of a relatively new guide to community-based fire management (CBFiM) issued by FAO. A whole five pages are devoted later on to a dreary recitation of four such workshops FAO held around the world between 2004 and 2009.

If you wanted to design a big new national programme of CBFiM, which I wouldn’t, then I suppose something along those lines might be an early step, but I would recommend starting with one small workshop and a trial project, before expanding on a province by province basis. More to the point, to assume that is what all readers of that manual want to do strikes me as a spectacular lack of imagination on FAO’s part.

A gem of an oxymoron that appears in the background section declares (my emphasis added):

“Sources of ignition and fuels are local; thus, the systems and frameworks of fire management are often best established at the provincial level, while monitoring and analysis are usually best dealt with at the national level. Yet discussion and debate often take place without reference to the appropriate scale of intervention.”

Quite! It continues:

“To ensure that suppression occurs effectively at the local level, that is, that unwanted and undesirable fires are kept small, everything else in the fire management equation must occur at higher levels, including effective coordination and cooperation of all fire management agencies.”

Putting out the fire in my backyard requires coordination and cooperation of ministry or sub-ministry agencies? I could maybe just about live with this tosh if the report was not supposed to be specifically about community-based fire management. Alas!

Some people, unfortunately, really just do not get bottom up. That’s all right; I don’t really get ballet. Thankfully, this need not greatly concern the world of ballet. If only we could say the same for community-based conservation and development.

How enterprising is your social enterprise?

So the SOCCKET ball is the latest incredibly dumb development idea to get the treatment from the blogosphere, led by tea-drinker Aaron Ausland. In his latest post he gives the SOCCKET ball co-inventor and ‘Chief Social Officer’ of Uncharted Play, Julia C Silverman, the right of reply.

“I’d like to clarify Uncharted Play’s business model for the SOCCKET product …  We are a social enterprise, not an NGO; we answer to our investors and are kept afloat by revenue, not donations as your post implies …

For our users in disadvantaged communities, corporations and public institutions underwrite the cost of SOCCKET distribution through bulk/wholesale ball purchases. Users (children) “earn” the balls by participating in the programming of our official NGO partners.

…

Our corporate partners are wonderful, but they are not development institutions. When they were deciding to work with us, they were evaluating whether to put marketing dollars into SOCCKET sponsorship or into another campaign, not another charity.”

So, let’s just be clear about this: Uncharted Play is ‘selling’ the SOCCKET balls, just not to the end users. I.e. they are relying upon a traditional aid model and CSR-style conscience-buying, and then trying to make money out of it. Sounds rather like some big ugly consultancies I know … More to the point their feedback model is as broken as the rest of the aid system. A bottom-of-the-pyramid business proposition this is not.

“Given the distribution of accountability, it would be all too easy for us to simply pay lip service to our social mission while dedicating the bulk of our financial and human resources to sales, marketing, etc.. However … this is not the case: we are truly focused on collaborating with communities to implement meaningful, catalytic programs, and – rather than resting on our laurels or focusing strictly on profit – we are taking aggressive action to engage closely with our partners and participants and track outcomes so that we can drive toward maximal positive impact.”

Really? This reads just like typical drivel from a development agency with negligible impact. What ‘impact’ are they seeking? How many kicks of the ball does it take for a kid to generate enough electricity to power a light while they do one hour’s homework? In fact it looks like the one part of their social enterprise that is doing its job well is the marketing department. So all very reminiscent of the NGO world …

I know plenty of people are very sceptical about the concept of a social enterprise. Uncharted Play unfortunately appears to be a living embodiment of all their arguments that it is just the same old guff dressed up in new clothes. A pity for all the truly enterprising social enterprises out there.

ps. Ms Silverman also claims:

“… kids have found the product to be truly magical. … When we actually say that the ball is special, that it can harness energy and power a lamp or a phone, there is always a collective yell of excitement.  Then, when we plug in a lamp to demonstrate, the kids’ eyes just pop out of their heads, and you can see the wheels beginning to turn.”

Has Ms Silverman considered that this might partly be because the kids have just received the coolest physics lesson of their lives? It would be great if more science lessons could be this inspiring, but I doubt very much it represents value for money for destitute kids.

Inflexion points

Various thoughts have collided in my head over the last week concerning value for money* in tropical conservation and development. I can see two possible inflexion points, which are clearly related in some sense, but also reflect different concerns linked to the issue of scaling up.

The first I’ll call the So What? point. At the bottom end of development, if someone spends $1,000, say, on donating school books, then most people’s reaction is simply to celebrate the fact that now some kids in a poor country have some books that they would not have had otherwise. It would seem churlish to question whether, in fact, there might be a better way to spend the money.

However, if someone else spends $10m on school books, then one is naturally inclined to question what is the eventual impact of this donation. How much better educated are the kids? How many are lifted out of poverty – or at least become less poor – as a result? They spent $10m on school books, but so what?

The second inflexion I’m calling the Get out of Bed point, and is the minimum grant size likely to interest a development practitioner. It is to a large part set by the organisation for which you work. BINGOs may not chase grants smaller than $1m, while the smallest NGOs will happily jump through several hoops for just a few thousand dollars.

The problem comes when a would-be donor perceives one as having a lower Get out of Bed threshold than one in fact does. One does not want to seem ungrateful, and maybe if there were less hoops  to jump through or less strings attached (hint to donors: these make small grants much more worthwhile), and/or if the grant would be easily renewable (dependable funding is always appreciated!) then we might go for it, but every application form takes up senior staff time when they might be better off just managing what they’ve already got on their plates. This can be a tricky issue for small but growing NGOs to navigate.

I haven’t had the time to really think this through, but I suggest that the best conservation and development action probably takes place when the two inflexion points are roughly aligned for both the implementing agency and its main donors. If the Get out of Bed threshold is much lower than the point at which the So What? test becomes applicable then the organisation concerned is desperately short of money (or fixated on raising money rather than achieving notable results). Conversely if  the Get out of Bed point is significantly higher than the  So What? question level, then the agency is probably swimming in cash and may well be more concerned about spending its great dollops of wonga than achieving value for money.

Contrary thoughts or suggested examples of the different cases are welcomed in the comments section.

* Note this discussion is not about the quality of service provided, efficiency of the activities undertaken, or the level of overheads applied. For the purposes of this post please assume such variables can be controlled for, although there may be an inevitable degree of confounding factors.

How many NGOs do we need?

Around this time of year one sees lots of ‘Away On Vacation’ signs up on blogs. Alas the reason for my lack of recent posts is just that I’ve been insanely busy. Plus a temporary internet outage for ~5 days interrupted my blog reading, and I’ve been struggling to catch up ever since. But I couldn’t let this piece from J of the ‘hood just pass by. He says:

“We need fewer NGOs.”

I suspect he is probably right, but I think there is a lot more to it than just a numbers game. I think we need both more NGOs and less NGOs. We need the useless ones to die as quick and as painless a death as possible, and then we need new NGOs to keep the existing ones on their toes. As David Week points out in the comments:

“We tend to forget that Oxfam, WV, STC and MSF all started out as corner stores, a meeting in someone’s living room, a zero-budget, zero-org with not much clue to begin with.”

J works in emergency humanitarian relief where the NGO scene is several orders of magnitude more crowded than that in tropical conservation and development, so my heart goes out to him for sitting in interminable cluster meetings in a vain attempt to coordinate who is doing what. I’m also broadly with him on the need for professionalism in conservation and development. If some random guy knocked on your door and suggested you send your child to a new school he was starting up you’d be mighty suspicious and have a list of questions longer than your arm, so why anyone would think poor people in developing countries should be any different beats me.

But for me it’s the whole attempt at coordination that is partly wrong. Not that coordination is a bad thing – we do it around here, and with a much smaller number of NGOs it actually works reasonably well – but there is a limit to what you can coordinate without strong metrics for determination of success and failure. Communism had lots of faults, but one of them was the fallacy that some bunch of bureaucrat planners in Moscow could effectively and efficiently oversee the economies of whole countries. In contrast, in the West for the most part we contented ourselves with letting the market sort things out. The market works on the simplest metric of all; profit. It didn’t always work perfectly (Betamax was a better technical solution that VHS), but it sure worked out better than the Soviet Union (no consumer video recorders at all).

The reason we have so many NGOs is that rich dupes keep on funding the incompetent ones. That they may then deliver the equivalent of the Soviet Union’s cars that no-one wanted to buy is as impossible for the donors to determine as it was for Moscow’s Politburo apparatchiks.

But it’s hard to blame the donors as most NGOs’ fund-raising literature is all much of a muchness; the same themes keep on cropping up and there is little basis on which the non-expert can use to choose. This works for the useless DIY aiders but also serves to protect the established BINGOs against effective upstarts (DIYers who actually hit on a better solution). It can take a long time to get much recognition in the Aid world; no from-nowhere-to-world-conquering-heroes like Google. Neither, apart from the satisfaction of a job well done, is there much in the way of reward for non-profit social entrepreneurs: BINGOs are not in the business of buying out their competition.

Too often it seems people expect too much from Aid / Development / Conservation, such that reducing their myriad outputs to one or more simple metrics would be almost impossible, ref my recent plea for simplicity in environmental certification. Simple metrics will always be distorting, but the merits of simplicity for comparing two rival service providers are substantial. Hence why I was most tickled by Sam Gardner’s recent vision of development finance in 2021, in which he posited the Sphere standards as the basis for competition amongst humanitarian relief agencies. Sam imagined this as being used by big institutional donors like DFID, but why not by the man in the street too?

How many NGOs do we need? As many as can effectively compete for rigorously apportioned funding! No cartel for the BINGOs and no room for crassly ignorant distributors of SWEDOW.

When “on the ground” is actually thin air

The REDD desk lists such “on the ground” initiatives as national REDD strategies and funding proposals.

Conservation and development agencies love to boast about their achievements on the ground. Though policy changes may be equally if not more important, nothing quite connects with the public at large as specific stories of actual conservation and development successes. These, however, can prove to be somewhat harder to pin down than marketing departments would like to admit, and so some fairly tenuous claims get magnified into complete myths.

I recall one MSc student recently visiting a nearby ‘project’. A big international NGO was involved and she’d seen some snazzy publicity leaflets talking about all the good work going on. The NGO had even boasted about its achievements at the Copenhagen summit in 2009. This project was a prime example of their good works, and therefore a ‘big thing’. Other international partners also appeared to be involved. All in all it looked an exciting subject for an MSc thesis. When she arrived in country, though, our heroine found that the substance behind these claims proved frustratingly elusive. In her memorable quote:

“The closer I get to this project, the less there seems to be to it.”

Eventually she reached the field site where despite over a decade of dysfunctional attempts to engage the communities, there were zero concrete achievements, and communities had no more rights or powers than they had twenty years ago. The guilty BINGO was actually a relatively late arrival on this scene, so cannot be entirely blamed for all these failures, but their marketing puff certainly was a signal distortion.

I don’t believe the BINGO ever meant to act fraudulently. In fact much of the publicity was generated early on in their project, and spoke about what the project was going to do; 2-3 years later any outside observer would tend to assume that at least some of these goals had been achieved. Naturally no BINGO is going to loudly announce such delivery failures.

Although in this case I think the BINGO concerned did not manage their project well, we should be realistic: not all conservation and development projects are going to be a success. NGOs, big and small, and other development agencies, who know this should not be so quick to rush out the publicity. That way, when they claimed successes on the ground we could be rather more confident that they really existed outside the marketing department’s fervent imagination.

Learning by Doing

Kudos to Owen Barder who has no lesser a dignitary on his Development Drums podcast than Tony Blair. Some of his answers are slightly evasive, suggesting to me that you can take the man out of front line politics, but you can’t take the politician out of the man. But he has also got some interesting things to say about his African Governance Initiative, and Lee the Bandit’s unerring sense for the hidden gem was fully functioning on this roughly transcribed extract:

“People often say to me ‘you’ve got to train the civil service of the country in order to be able to do the things they need to do.’ I personally think you can spend literally hundreds of millions of dollars doing that and nothing much come out of it (Barder: and we do). What we do is different in 3 crucial respects, the first is we combine a political interaction … the second is we prioritize, this is about delivering programs … people have this view that if you train up the civil service then they can deliver the programs. My view is that if you work on delivering  specific prioritised programs, you will get out of that the capacity that you require and can work on for delivering other things, and that its in the practical prioritization and doing things that makes the difference. The third thing is that our teams live in the country, they work alongside their counterparts in the country, there is a very strong interaction.”

Later on in the interview Tony Blair uses the actual phrase Learning by Doing. Three cheers say I, for this is definitely something in which I believe, big time.

Traditional development approaches to capacity building tend to focus around short training courses. These can teach people technical skills but they fail to stimulate the critical thinking that is essential to solving real world problems. Instead we get dysfunctional institutions that superficially look capable of doing a job, but lacking the internal engine to make it tick. This problem applies equally to government institutions, local NGOs and even local branches of international NGOs.

Learning by Doing gets people to work through an entire process, and in doing so develops a whole host of soft skills. In practice this is achieved through mentoring and is inevitably a slow process that is not readily susceptible to rapid scaling up. Indeed this lack of scalability is, I believe, a major factor hamstringing attempts to transform pilot projects into large national programmes. Development agencies have attempted to get around this problem by the ‘Training of Trainers’ approach, but you have to have trained some really top notch trainers if this is not to suffer from the inevitable Chinese-whispers-style degradation of skills imparted. Any way, nobody I yet know has tried ‘Mentoring of Mentors’.

One big problem with this approach can be the receptiveness of the institution whose staff are being mentored. Tony Blair’s initiative appears to work at the very highest levels, and to be aimed at supporting African presidents who really want to get things moving (even if this is at the expense of democratic accountability). He takes an admirably realistic approach that focuses on just a few priorities over a presidential term of office. Elsewhere in developing countries, however, just about every institution of government has multiple sources of donor support, many of whom will be engaged in some kind of capacity building. It is far from clear that the staff from these institutions are interested primarily in delivering change, or more in the per diems and other ephemeral benefits. I gather that the ‘traditional’ Technical Adviser role is losing popularity in favour of short term consultants precisely because developing country institutions find these less intrusive.

So three cheers for Learning by Doing and Mentoring. And thirty three cheers for institutions who are open enough to genuinely want it!

Who you gonna call?

Ghostbusters: just more likeable than the Federal Bureau of Exorcism.

Musing further on  my post yesterday on the beauty of small projects and human resources constraints, it occurred to me that I left out an important additional consideration. It’s a major advantage of NGOs over government agencies: not only are their staff likely to be better trained and more flexible, they’re not also tasked with enforcing the law.

This issue occurs especially in community conservation projects.* Twenty years ago, if you were a wildlife, forestry or marine biology geezer working in a developing country then chances are you worked for the government. So when the community conservation revolution broke out many advocates naturally assumed that they’d have to retrain the government officials.

This is certainly not impossible; I’ve worked with some excellent current and ex-government natural resources officials who are capable of establishing great relationships with local communities. But I think they are more the exception than the rule. Picture yourself as a poor farmer: yesterday a game ranger turns up to arrest your neighbour for a naughty spot of poaching in the nearby national park, then today he turns up all smiles to preach to all of you that the national park is your friend, and that everyone can – and should – work together peacefully, and for the betterment of all. How are you going to feel?

Trust is absolutely central to successful community engagement in conservation. Despite our various more concrete achievements in the projects I help to run, establishing trust is one of our successes of which I am most proud, and frequently find myself returning to it when people ask us what they think is the secret to our success. Without trust, you’ll find community members just turn up to meetings for the per diems, with trust you’re often preaching to the choir (though they still appreciate those per diems, thank you very much).

I can readily appreciate how different the tropical conservation landscape looked 20 years ago, and that there might have been little alternative to working with government agencies and staff on community conservation. But with plenty of alternatives now available (in most tropical countries) I think a reappraisal is called for: if we were to design a system from scratch we would never dream of combining the role of policeman and community facilitator.

Donors should take note, and place themselves in the position of rural communities wanting to engage in conservation. Who you gonna call?

* Although I think the wider principles are more generally applicable to other community development projects: poor farmers may not have much respect for government-employed agricultural extension workers.

The Scaling-up Fallacy

Last month Justin Sandefur at CGD lamented the regrettable failure of the Kenyan government to sustain a successful school-based de-worming programme after donor funding was withdrawn due to corruption in the Education Ministry. This is another good example of the sustainability paradox: despite clear evidence that this programme was extremely cost-effective it was cut when the donor funding was withdrawn. I assume that this was as much a political act intended to hurt the donors – who lost something they cared about – but as such is clearly rather callous. But, more than anything, it is another example of the phenomenon that what the donors want and what the recipient country government want are often not the same thing.

This, however, is not exactly news around here. More interestingly Sandefur also suggests that this raises questions about “the feasibility of turning small NGO pilots into manageable national policies”, although he failed to elaborate much on that idea in the rest of his post. This is something I’ve been thinking about a bit recently, and I think there is an important additional argument to be made here.

Whether a pilot is being developed by an NGO or a bespoke, direct donor-funded project, it will have its own management structure. It will also have a significant investment of technical advice and support that is inevitably diluted when a project is transformed into a national programme. However, I can live with that; if we want aid to be cost efficient, then we need to be able to realise economies of scale on techniques that have been shown to work.*

My beef is with the management. Because, to the international aid industry, scaling up nearly always means launching a nationwide government programme. In doing so the donors discard the effective management that produced the initial successes in favour of a dysfunctional government bureaucracy. Not only do you lose some basic management nous, but you also lose the driving vision, the leadership that got the pilot project to where it did.

When Larry Page and Sergey Brin founded Google, they didn’t show some initial promise and then hand their genius idea over the government. Instead they secured some outside investment including big business management expertise (Eric Schmidt) – thus addressing their ‘absorptive capacity’ – and grew the company to the multinational search behemoth it is today. More to the point, Google isn’t just big; it continues to be incredibly successful.

I’ve blogged before (here and here) about the importance of the quality of management in delivering conservation and development results. The standard donor approach to scaling up suggests that donors remain stuck in a rut that emphasises technical barriers (leading to misdiagnoses of project failure) over management constraints, combined with the belief that all you need is a bit of capacity-building in profoundly dysfunctional institutions to turn it around.

The next time donors are seeking to scale up a successful programme, I hope they will remember the Google story, the Grameen Bank story, and the countless other examples of private sector efficacy in turning innovation into successful business models. After all, most donors are capitalist countries, not socialist ones, and there’s a reason that communism collapsed.

* There is another argument to be made here that many projects are scaled up before the jury has properly returned a verdict, leaving key issues still unresolved. But, conversely, if an approach does appear to be working, I can understand how funders, desperate for new solutions, may pile in prematurely.