Theorising pig flight

“Everyone these days (funders, bosses etc) seems to be demanding a Theory of Change (ToC), although when challenged, many have only the haziest notion of what they mean by it. It’s a great opportunity, but also a risk, if ToCs become so debased that they are no more than logframes on steroids.”

That was Duncan Green writing a couple of months back. I totally dig the turn of phrase, but (luckily!) have so far escaped any such experiences of being enslaved to a donor’s preconception of what a ToC should look like. On the other hand I do find logframes (or ‘lockframes’ in the memorable corruption) more than a bit tiresome such that I might be inclined to reverse the comparison, and describe logframes as theories of change on steroids.

If you are worried that you might fall into that class of people who “have only the haziest notion” of what a ToC is then you can go read Duncan’s blog post (plus an excellent selection of comments) or Google for a whole bunch of other informative web sites. But over in this little corner of cyberspace I quite like my ignorance of the more formal definitions. Not that the above sources are not useful, quite the opposite, but I prefer the ‘pornography test’, which is to say I believe I have a pretty good intuitive idea of what a theory of change looks like, and I reckon I know one when I see one.

To me a ToC is primarily just a reasoned explanation of how what one proposes to do will actually deliver the impact you expect to achieve. It can be summarised in nicely boxed flow diagrams and the like, but for me the real test of a ToC is that it can stand up to reasoned, sceptical argument.

Where, I believe, so many conservation and development projects go wrong in their design, is not in their use or lack of use any particular framework, but in just plain sloppy thinking and lack of self-criticism. Part of the problem in development, it seems to me, is that we are too often too nice to each other, and not inclined to criticise (constructively!). This challenge can be exacerbated when discussions cross cultural boundaries: local ownership and deference to regional social norms are important, but should not trump having a workable plan in the first place.

Conversely we are also often too formal. A single written proposal, however, well constructed is never as satisfying as being able to discuss and probe people’s plans in person. And who reads those tediously long project documents any way? The result: too many projects approved primarily on the basis of the executive summary, without real testing of assumptions. And that’s when those flashy graphics really come into their own: great for communicating the central thrust of an idea, useless at exposing logical fallacies.

MJ’s theory of porcine aerodynamics: flashy graphics may not stand up to serious scrutiny.

So I like donors who are prepared to get into a real conversation with their grantees, to get to know them and their plans a bit better. Such relationships can more easily support adaptive management, which in turn allows you to be a bit more relaxed about any flaws in the original proposal, because now you have a framework in which to manage deviations from the plan.

And how do you succeed with all those awkward discussions in which design flaws are impertinently probed? As one of the commenters on Duncan’s post put it: “the first order of business is to build TRUST.”

Advertisement

Donors: show us your scorecards

A few months ago Lant Pritchett hectored the World Bank Executive Directors to publish their scorecards on the three candidates for the WB presidency. It was a nice idea which unfortunately didn’t get a lot of traction, presumably because WB presidential selection is essentially a political process, despite all the publicly claimed desire to simply find the best candidate.

But why limit this idea just to the WB president’s selection? A lot of aid funding for NGOs comes through competitive calls for proposals. Some donors, the EU comes to mind, define tediously exact criteria by which proposals will be judged, others are rather woollier. Wooliness doesn’t have to be a bad thing – it allows extra weight to be given to proposals which contain that hard-to-define X factor – but for all the fact that judgements are necessarily hugely subjective – often favouring the known, established applicants – precision has a lot of merits in clarifying ranking of proposals. When the number of proposals is high it very probably becomes the only way of effectively comparing the full range of applications, so I assume most donors practice something along these lines.

So why not publish the results? In the age of openness and transparency, why are donor decisions shrouded in mystery? The EU may tell you the score allotted to your proposal, but do not tell you the threshold that was necessary to obtain funding. (When your proposal has averaged over 80% it does leave you wondering exactly how good it needs to be?)

Donor decision-making needs to be final, there is no question of that. Endless appeals against the system, however it is rigged, won’t do anyone any good. But it would still make the lives of NGO fundraisers easier if we knew both what had and had not been successful in the past, and what were the various scores. (It might even save us from writing the odd wasted application.) So donors, how about a bit more sunlight on your decision making? Go on, show us your scorecards!

Facilitating What?

Aaron Ausland raises the age old question of how exactly ‘participatory’ all this community-centred development (and conservation) work is. It’s a serious and almost inescapable problem. Our overall goal might be facilitating community development, but in any given meeting we are almost certainly focused on facilitating the next step of  our project. A good facilitator will ensure a meeting stays reasonably focused on the topic at hand, but, given the topic was most likely chosen by the facilitator, how can we be sure this isn’t just a box-ticking exercise? Is this what the community really most want to discuss? Probably not …

I think this issue is another facet of paternalism in development.  Aaron’s criticisms of ‘facipulation’ as a ‘bad thing’ are 100% on the money, but I also think a certain amount of facipulation is almost inevitable, and even sometimes desirable, because the fullest forms of participation are just too onerous. (The cost-benefit ratio of the project disappears to infinity.) Sometimes a certain amount of facipulation may also be appropriate in order to achieve a constructive outcome from a meeting with local politicians or officials who might otherwise cause trouble.

Another difficulty is that if we are to achieve the highest levels of community participation, then we have to be prepared to let our projects take a very different direction than what we perhaps first envisaged. This is often problematic since if we’ve promised our donor a chalk project we cannot then deliver a cheese project, even if that’s what the communities want. Even if we were wise enough to promise donors a mixture of chalk and cheese in the first place, high levels of community control and direction inevitably pose management challenges. This requires high calibre staff on the ground able to adapt and adjust strategies on the go.

In all of this what I think it is very important is to realise when one is guilty of facipulation; to understand when the reality falls short of the proposal rhetoric. Knowing this, we will hopefully strive to ensure the manipulation part is minimised. In short, a guilty conscience is good for keeping us in check!

Bad news brings in the wonga

We’re all familiar with the unfortunate fact of life that bad news tends to be much more newsworthy than good news, e.g. this recent example.

This is doubly true for NGOs, as Karen Rothmeyer points out. (H/T: Tom Murphy) We depend upon bad news to bring in the money. So, although we do like to trumpet our successes, the mantra is always about how much more there is to do.

Got a choice of statistics to use? Just use the one which portrays matters the worst, and a molehill can be turned into a mountain over night. Have I ever done this? Yes, although I don’t think ever as badly as the case of the inflating population statistics for Kibera slum in Nairobi that Rothmeyer reports. Do I feel guilty? A bit. Would I do it again? Probably, since all our competitors are doing it, we need to do so too if we are going to continue to receive funds for our programmes. This is one time where being principled doesn’t get you very far.

It is a tragedy of the commons of information: we’d all be better off if everyone was scrupulously honest, but at the individual or organisational level the incentives are all in favour of scare-mongering. Africa, and other supposedly ‘impoverished’ places are caught in the middle: impoverished of good news stories about them.

The same is true in conservation: the IUCN Red List catalogues all species that are “facing a high risk of global extinction” (description provided by Google), and yet its lower risk categories, Near Threatened and Least Concern, cover species that are not exactly at a high risk of extinction. Still, I regularly hear people bandying around numbers of ‘red-listed’ species in their particular habitat, and I always wonder how many of them are classified NT or LC.

This bias towards bad news, I think, can be particularly problematic when it drives global prioritising and funding decisions: the result is that money ends up chasing problems rather than solutions.

Good strategies need good implementation

We’ve just submitted another funding application. As is usual in such cases I took on the lion’s share of the writing. Indeed this is one of the most important roles played by me and my peers in other small NGOs about the developing world.

We’re the rain-makers for three reasons. Firstly our command of English (or French) is usually better than our local counterparts: this becomes very important in condensing a 110 word summary of your proposed project into the 100 word maximum allowed by the donor on the application form. Secondly, the people making the decisions are usually quite similar to us (similar backgrounds etc.), so we have a better idea of what makes them tick, and thus how to hit the right buttons (succinctly!) in our proposal. Thirdly, expat technical advisers tend to be the key strategic thinkers / vision propounders: our local colleagues keep us grounded (extremely important) but the (elite) Western education that many TAs have enjoyed seems more effective at providing this kind of input than local ‘teacher always knows best’ rote learning. None of which is to say that there are not local counterparts who are extremely good in these skills, but usually they’ve had the benefit of a Western education, at least at university level, and hence are relatively rare.

Larger NGOs have realised this key role of the TA as rain-maker, and, following basic economic theory of the benefits of increasing specialisation, employ specialist proposal writers. The more junior of these folks are just there to do the translation of good ideas into donorese, but the more senior are brought in to actually provide the project conceptualization too. Other BINGOs habitually use consultants for this role, with a similar effect.

The trouble is that even the best strategies turn out to have flaws. As the old military saw goes: no battle plan survives contact with the enemy. When things go wrong, as they inevitably will, you need to have a firm grasp on the big picture ‘vision’ for the project in order to work out how to address them. This becomes even more important when there is no big blow-out to tell you that things are going wrong, just a steady failure to achieve any impact: all those field visits may individually appear to be successful, but if the changes you are trying to instigate do not happen, then overall success is negligible. You need staff with good management skills to recognise these kind of situations, and adapt appropriately. You also need the management culture to encourage escalation of such concerns rather than just blindly following the agreed activity plan.

Unfortunately, in BINGOs, this often does not happen. “X came up with a great plan, but all the field team did was drive up and down the mountain a lot.” is one summation I heard. Mid-term reviews are supposed to help correct such situations, but the evaluating consultants only have so much leverage and are only around for a short period of time. On paper the BINGOs claim to follow all kind of best practice in planning and adaptive management, but the reality is that you need high calibre staff to implement this. At the end of the day, I think, there is no real substitute for having someone close to the field team with a clear understanding and ownership of the strategy; someone who can take responsibility when things go wrong and ensure the project gets back on track, and damn the work plan!

I recall once reading an interview with a multi-millionaire self-made businessman. The interviewer asked him what he thought made him so successful. The guy replied that he reckoned pretty much anyone could come up with a good business idea, the real trick is in making it work, the commitment and attention to detail that requires. I think the same is just as true of conservation and development (although it’s harder to come up with the good ideas): good strategies need equally good implementation.

My ideal donor

I just received a delightful email from a chap who was setting up a new grants scheme for somewhere in Sub-Saharan Africa. He had read my rant entitled Preposal Prepostery, and wanted to know what advice I would give to a donor wanting to do the right thing. My dream come true: a chance to design the ideal donor!

Here are ten tips I gave him. Please suggest more in the comments, and I’ll pull the best together into a second piece.

1. Offer funding that is presumed to continue (unless there is a bad evaluation or something) rather than presumed to stop (but where you can apply for follow-on funds). Obviously your ability to do this will be constrained by your own funder(s) – you don’t want to make promises you cannot keep – but maybe this is the kind of innovation which might make you more attractive if sold to them in the right way. This doesn’t mean promising to fund projects in perpetuity, but there should be a sensible strategy to achieve long term sustainability (handing over to the local government does not count!) either because need for the external assistance will eventually subside (e.g. farmers will have learned the new techniques being promulgated) or because alternative funding can be found. But these should be long term issues to be reviewed and gradually refined over the length of the project, not something to be dreamt up for the initial proposal.
2. Give 100% funds upfront at the beginning of each year. If you feel the need to incentivise report production, do it through personal perks like invites to lesson-sharing workshops. Never let grantees suffer funding gaps!
3. Keep budgets short. I reckon 10 different budget lines should generally be enough. Don’t demand ridiculous levels of detail in financial reporting, but do demand proper audits. Even ask to be consulted over the auditing process so you can ask auditors to look for specific issues of concern to you. (You might have to pay extra for that.)
4. Don’t get too hung up on large proportions of budgets going on certain things. Maybe the grantee needs that. In particular a high % going on salaries might mean they are employing high calibre staff which is about the best indicator of success I can think of.
5. Consider funding 6 month inception phases as a matter of course. After 3 months the grantee should submit a full project document setting out their plans in a lot more detail. This can be further negotiated, until satisfactory. Funds release beyond the 6 month inception phase is dependent upon a satisfactory agreement over the detailed project document. This can be repeated every year so long as the grantee has at least 3 months funds carry-over so they can keep on working in the meantime. Major asset purchases should be held off until after the inception phase. Inception phases are really useful where you’re not sure about an applicant: maybe they just don’t have the writing skills to prepare a good application, and you can offer support to the applicant to this phase, assessing for yourselves what needs to happen. Conversely don’t demand highly detailed proposals before the inception phase.
6. Require applicants to complete a risk analysis together with how they intend to address these risks. This should be reviewed annually.
7. Focus on impact / outcomes over outputs / immediate objectives. Grantees should clearly communicate what’s important; that way when things go wrong (as they always do) you can hope they focus on the impact and not on which outputs they promised to deliver.
8. Beware of box ticking exercises. A logframe is no substitute for a clear strategy, and if you have a clear strategy the log frame is relatively unimportant. (Tho some people may find it useful in both formulating their strategy and communicating it.) Equally, SMART objectives are all very well but sometimes can get a bit silly, e.g. time-bound should simply be assumed to be by end of project unless otherwise stated. On the other hand they do force the indisciplined to get properly specific.
9. Allow for free-form proposals with some suggested headings, making it clear applicants should feel free to add their own, and exclude any that are not applicable.
10. Be very careful about your call for proposals: restrictive calls simply force people into straitjackets, writing proposals for what you want to fund rather than what they want to do. Even quite general restrictions can have this effect. E.g. I recently review applications for a small conservation grant which required applicants to include a local communications / outreach component. There were some good proposals which suffered from having this bolted on where it didn’t make full sense.

And here’s an eleventh I just thought of:

11. If you must demand certain issues be mainstreamed, then allow applicants to mark them as not applicable without that being a black mark against them.

Proposal Prepostery

We recently got a consultant’s report appraising a project proposal in which we’re a partner. The good news is that the donors are going to fund it, but they want a revised proposal. Apparently some fairly important issues were not clear to the donors which is fair enough. However, appraising consultants clearly believe they’re not earning their (very generous slice of) daily bread if they don’t write a long, involved report. So we’ve got a loooong list of issues to respond to, some of which are plain daft*, some querying things we were asked to do by the donors after they received the first draft, and others just go to show the consultants didn’t spend enough time checking issues, rehashing old concerns that were addressed a while ago.

But the biggest problem we had with the consultants’ approach was that it is VERY BUREAUCRATIC. We are constantly being told that they sense potential, innovative thinking amongst the partners etc., but then complain that it is not fully reflected in the project proposal. This seems to me to miss the wood for the trees. Surely, the point about this exercise is not to produce the perfect proposal, but to get the best project possible. If they think the partners have the right ideas who really cares about the project proposal? The consultants’ approach was exemplified in the following recommendation:

“Writing proposals for funding, and delivering to donors requirements, is an acquired skill, requiring recognition of the donors’ needs for accountability and monitoring. This proposal has clearly worked hard to achieve the necessary standard but some gaps remain. External support appears necessary to complete this proposal, and would also help [the  lead partner] and the partners to better manage future donor applications, reporting and partner relationships.”

Eh? Let me just get this right: you’re recommending we spend further lavish sums on you and your ilk so all the donors can wow themselves at our amazing use of the latest jargon and buzzwords? There was plenty more along these lines, e.g. suggesting one of our partners lacks the wherewithal to chair a meeting neutrally (so the consultants could do it for us!) which is just utter tosh.

On the other hand, we do have reason to be concerned as to the consultants’ expertise. One has negligible previous experience of the country in which we work whilst the other has never been involved in a project remotely like the one we were proposing.

So let’s be clear about this: if we want someone to give us some money then we need to explain clearly what it is for, that is self evident. Furthermore they have the right to hire an independent external evaluator to check on their behalf for any red flag issues. But, if a project has taken us months to plan, and incorporates lessons from a previous pilot phase, funded by one of the same donors, then in one week’s running around the country you are unlikely to understand every aspect sufficiently to offer detailed criticism. And it’s just plain rude to try making extra work for yourself when you’re paid so much more than the real innovators. Advice is welcome, self-promotion  and unfounded criticism not.

However, an equal responsibility lies with the donors who are responsible for the context in which said consultants are operating; if they didn’t demand such ridiculous levels of paperwork the consultants wouldn’t write reports like this. Donors could give more freedom and responsibility to in-country (i.e. embassy) staff to approve and support projects which they know are good without endless proposal tinkering. (Though for this to really work well, said embassy staff would have to stay for longer stints than 2-3 years.) They should also stop trying to influence small details in such projects (the consultants advocated a role for the donors on the project steering committee which is absolutely not going to happen), and instead have greater confidence in those they have chosen to fund. (Give them the rope to hang themselves if so be it.) No wonder developing country officials get so exasperated with donor meddling.

Sadly, I cannot imagine this situation changing very fast, so it’ll be back to that proposal in the new year …

* I’ve previously written about gender mainstreaming and conservation. Now, apparently, we have to mainstream health issues too despite the complete lack of relevance to the proposed project.

A useful lesson

When I was about 15 I had an English teacher who didn’t much care for dissenting views. A critical or negative essay about one of our set texts would be marked down as, or so it seemed, showing insufficient appreciation for the magnificent literary powers of the author. In quasi-Orwellian fashion we were supposed to like whatever it was we were given to read. Conversely, I found, if I wrote an essay praising the text and full of flowery phrases like “innate symbolism” (still damned if I know what that one meant, >20 years after I wrote it) then I got an A or A+ regardless as to whether or not my essay corresponded to my actual views on the book in question. At the time all this did was put me off all literary criticism and associated arts studies for the rest of my life (so far). Years later I regard it as one of the most important formative experiences of my school years, and critical to resolving quandaries like the need to mainstream gender issues into a proposal that has little to do with gender.

ps. For those who like their critics not to hold back on the sarcasm, the full-frontal demolition job on the **** one has to put into funding applications is here.

pps. Actually I liked the Orwell!