Scraps from big brother’s table

“How do you find working with [name of BINGO withheld]? How could the relationship be improved?”

These questions, and others in a similar vein, were asked of me and my colleagues last week by consultants hired by a major donor of a BINGO that supports us with some of said donor’s money. The donor wanted to see whether they were getting fair value for money from the relationship.

Equal partners, right?

As with so many other ‘partnerships’ in the aid world, the relationship between a BINGO and small local NGO is far from an equal one whatever anyone claims. Indeed, just like recovering drug addicts, so for BINGOs: the first step in dealing with the problem is to recognise that it exists and that simply mouthing endless platitudes will not make it go away.

There is also no avoiding the fact that societal pressures (aka donor preferences) mean that both sides, the BINGOs and the local NGOs, are addicted to such relationships, with little hope of being weaned off. Moreover, for both sides there is often very little choice in who they partner with: at least in tropical conservation BINGOs largely try to stay out of one another’s territories, whilst vice versa the choice of capable local partners for a BINGO to work with may often be quite small. So to a significant degree partners are stuck with one another.

A more reflexive, self-critical approach will certainly help; being honest about the nature of the relationship does not have to mean being arrogant about it. As with so many things in life, that understanding, personal touch can make all the difference to the relationship. But is there anything more that could be done?

Ultimately I think it comes down to a question of how and to whom the BINGO are accountable. This echoes one of the main themes in development aid in recent years to which a leading response has been a call for more transparency, and the emergence of things like the International Aid Transparency Initiative. I think the same trick could work again here.

One of the things we found frustrating about our relationship with the BINGO was being drip fed small amounts of money that covered only a few months and were then micro-managed, e.g. budget lines of only $1-2,000. (The BINGO did not have systems that could differentiate between capable, trusted local partners, and others about whom they were more cautious.) The micro-management was a pain, but in many ways it was the drip-feeding of funds that caused the bigger problem, because they were almost impossible to plan for.

Partly, I think, the BINGO did this to remain in control, but partly also I think because it kept us in Oliver Twist mode – always asking for more but rarely seeing the bigger picture – and thus allowed them to prioritize their own management costs and strategies. I would like to see BINGOs being much more open about their own funding streams; how they are divvied up, managed and reported. The middle-man role is an important one, and for our part, those of us in smaller NGOs need to recognise that performing this role does not come cheap.

So when a big chunk of the donor money goes on BINGO bureaucracy (no doubt disguised as technical assistance) we shouldn’t bleat too much. Proposing standards for what is an appropriate slice to take is likely to be counterproductive (in the same way that misdirected focus on overheads has proven). Instead the BINGO becomes accountable to some degree to their small NGO partners (who are already have to do lots of accounting to their bigger brethren), and both sides of the partnership can better evaluate whether or not they are getting value for money from the relationship. Longer lasting commitments of funds would also help the smaller NGOs to plan better.

Advertisement

M&E – a top-down imposition

Yesterday I promised you some reflections on Prichett et al.’s working paper on improving monitoring and evaluation. They correctly identify that rigorous impact evaluation works too slowly for most management purposes, costs a lot (putting it beyond many project implementers), and is often highly context specific so not readily generalizable (the standard critique of randomized control trials).

Their proposed solution is to insert a second (lower case) ‘e’ that stands for experiential learning: MeE. In particular they propose that project managers should have considerable freedom to adaptively manage their project, and moreover should be encouraged to try different approaches to see what works best, thus avoiding the problem of over-specificity that I highlighted in the quotes I pulled out in yesterday’s post. They suggest that this will give much more like the real-time information that project managers need, as well as exploring more cost-effectively different project designs, than establishing a separate RCT for each one. (Which may not be sufficient any way, as which design is optimal may be context-specific.)

It is an excellent paper, and their proposal has a lot to recommend itself if you work for a big development agency. But, I cannot see it working very well at the small NGO end of the market where I operate. The problem is not really specific to experiential learning as to the whole gamut of impact evaluation as it applies to project design and management amongst small NGOs. If I think about the best small NGOs I know and have worked with, several features are often apparent that reduce the incentives for impact evaluation:

• Small NGO types tend to be ‘doers’ rather than ‘thinkers’ – given the choice we will nearly always invest more money in implementation than M&E.
• Many small NGOs have fairly modest aims which do not need sophisticated M&E to assess.
• Other small NGOS are of the nimble innovator types. They may be iterating too rapidly for it to be easily captured in M&E, and do not have resources to iterate in such a systematic manner.
• Such NGOs have the capacity to learn very rapidly internally for relatively little investment of time and effort; there is no big institutional ship to turn around. Instead, for these small NGOs new learning leads rapidly to more impact and the potential for more learning.
• In contrast clearly analysing and communicating these lessons can involve a significant investment of effort that does little (except perhaps to support fund-raising) to deliver better results on the NGO’s chosen impact bottom line.
• Thus generating new learning and achieving greater immediate impact can be much cheaper for such NGOs than disseminating lessons already learned.

Donors and small NGO partners obviously have a role to play in helping offset this tendency (which is not always 100% healthy), but, as I have remarked before, there seems to me to be an inherent contradiction in the calls both for bigger/better M&E and nimbler project implementation in an attempt to mimic the rapid success of internet-era start-ups.

The contradiction becomes more apparent when one realises that while business may regularly monitor all manner of variables that are relevant to their business (e.g. page hits as well as sales), they always have an instant answer when you ask about their ‘impact’: it’s the size of their profits. No construction of the counter-factual required there!

I also suspect that few aid beneficiaries care much about what any impact evaluation may or may not say so long as they are getting good results out of the project itself. Thus it becomes clear that much M&E, and certainly impact evaluations, are essentially a top-down imposition by donors understandably keen to know the results of their funding, and at odds with the bottom up approach many people in development advocate.

So the real question is: does the donor and wider-development community get value for money from the impact evaluations they demand? This is a question that Prichett et al. raise several times. The answer seems to be related to the challenge of scaling up, a relentless pressure in a lot of conservation and development work that I have repeatedly queried (e.g. see here and here.) I.e. impact evaluation and Prichett et al.’s experiential learning is all about learning how to move from a successful pilot to national programmes and similar projects in other countries.

Here I return to the internet start-up analogy. Did Google get where it is as a result of an impact evaluation? No it grew organically! If you want more bottom up development, which this blogger does, maybe the solution is less evaluation and more of a market-based approach in which successful implementers are simply invited to replicate their successes along the lines that I suggested yesterday?

Now before I chuck the whole M&E set overboard, a few basic points need to be made in return. Firstly, and most obviously, claiming ‘success’ is easy when all you need to do is check your bank balance. Determining which pilot projects are successful is not always so straightforward – although not always as difficult as might be supposed – and essentially requires some kind of impact evaluation. Indeed the converse problem often arises of a new fad rapidly gaining popularity far faster than evidence of its efficacy: the micro-lending boom comes to mind. And as those classic RCTs around improving educational attainment in Kenya show, sometimes it’s not so much about what is successful, but what gives the most success for your money. Indeed, Pritchett et al. lament the demise of project ‘valuation’ and computation of value-for-money metrics by large development agencies.

I conclude that idealists who want all their development to be 100% bottom up are living in cloud cuckoo land. Even if we dismantled the whole international aid industry, governments still regularly engage in this sort of thing within their own countries, often under pressure from their own electorates. So if the people want development aid then the paymasters are going to need to some evidence on which to base their decisions. Most of all, what this humble blogger would really like to see, is donors actually paying attention to these things, instead of continuing to commit large chunks of funding to projects and programmes they know are doomed. Better to over-fund something that has a decent chance of success than flush your money down the plughole of the utterly unfeasible.

Are donors getting value for money from the impact evaluations they demand? Only if they act on the results!

Small still is beautiful

The beautiful bee hummingbird is the smallest bird in the world.

Last year I blogged about how I believe that in conservation and development small is often beautiful, remarking:

“small projects and organisations are a lot more personal; I think this element of personal endeavour can do a lot to ameliorate the charge of development work being patronising.”

I stand by this, and I think the recent small kerfuffle over Community-led total sanitation (CLTS) is as good an indication of this as any. First, Robert Chambers, who I genuinely respect, praised it in a piece of on the Guardian and Duncan Green’s Poverty to Power blog. On that blog I posted a comment querying how well this translated from a small pilot project into a big donor-funded programme (see also my recent post on the Scaling Up Fallacy):

“It’s great to hear of new bottom up approaches like this. I have a question about roll out … I can see how this kind of approach could work at a pilot level with well-trained *sensitive* facilitators. But what about when it is scaled up? How do communities react when a local govt health official comes along and shames them all? From my experience I can see how the whole approach might suffer from institutionalization – and the different relationship people tend to have with officialdom – but seek enlightenment from my cynicism.”

Alas I received no enlightenment.* Then Liz Chatterjee responded with a comment piece in the Guardian that raised many of the ghosts that I had speculated at. Whilst I should be cautious about assuming that cause and effect align with my own prejudices, I cannot help thinking that at least some of the more degrading aspects of CLTS might be alleviated in a pilot project, and/or where you have a small NGO with dedicated, caring staff who are alert to the possibility that things could go wrong.

Big organisations, and especially big government bureaucracies, are inherently clunkier than their smaller, nimbler cousins. They tend to resort to top-down, even when implementing a supposedly bottom-up approach. Staff subjected to a couple of weeks’ training will try to adhere to a perceived blue-print, rather than have the confidence to innovate and respond flexibly to the impacts they achieve in the field.

Unfortunately, these conclusions are pretty depressing for anyone expecting international aid to have large-scale positive impacts. It suggests big economies of scale are really hard to achieve. And I think this is true of most community development oriented projects. More than anything else these depend on the quality of the staff implementing them, and the talent pool in developing countries can be worryingly shallow.

But Aid can and does support many other types of projects which may be more susceptible to rapid expansion. If donors have the right kind of relationship with the host country government, as Owen Barder implies with many of his posts on aid successes in Ethiopia, then maybe you can improve services at a larger scale. That is not to say that we should give up on community projects – quite the opposite (as otherwise I’d be out of a job!) – but we need to understand their limitations, and who are the best agents to support them.

* Hint: even if you are an aging giant of development studies, if you’re going to use new media, then you should also make an effort to utilise its capacity for rapid interaction with your readers that more traditional academic publishing largely lacks.

Small is Beautiful

J over at the Tales from the Hood blog has recently been reminding us about how important professionalism is in development (1, 2, 3, 4). For the most part I agree with him: I find it incredibly frustrating dealing with well-meaning amateurs whose suggestions are mostly the opposite of helpful. But I believe that the drive for professionalism (including proper professional standards) needs to be balanced with consideration for what can be lost through taking such a focus.

Exhibit number one is to note that all the ‘professionalism’ of the donors, multi-lateral agencies and the big NGOs has not got us very far to date. The conservation and development industries may well have made life more bearable for millions of poor people around the world and mitigated some of the worst environmental practices, but both have fallen a long way short of all the promises they made.

It is true that many flaws of international aid have been pointed out by various commentators over time (see my blog roll for a small selection), and perhaps if all these flaws were addressed, the professional approach of all these various agencies would suddenly bear more fruit. But right now, we don’t know that for sure.

What does seem clear to me is that ‘professionalism’ generally seems to be associated with the established players who have the resources to hire the right people and do all the proper evaluations before embarking on a new course of action. I think this omits an important class of conservation and development initiatives.

Small is beautiful. I know it is a cliché, but there is a lot of truth to it. Firstly small projects can be a lot easier to manage; lack of complexity is certainly a virtue. Secondly small projects and organisations are a lot more personal; I think this element of personal endeavour can do a lot to ameliorate the charge of development work being patronising.

Small also allows for experimentation; where the sums are low, there often won’t be much lost if a project collapses for having failed to follow one or more pieces of best practice. Some of these holes can, and should, be filled in later before scaling up (if that is the goal). Bill Easterly constantly reminds us of the power of many different people making their own separate attempts to achieve their goals over a centrally-planned system. I think the aid industry could benefit from a lot more disruption from nimble, radical-thinking start-ups.

All in all, despite the manifest problems of DIY aid, if I had a donation to make, I’d far rather give it to a small local organisation I know well with relatively modest objectives and a long-term commitment to the communities it supports than to a BINGO.