It is a truth universally acknowledged that people hate filling out forms. From tax returns to job applications, forms force us to represent ourselves in ways decided by anonymous bureaucrats who know nothing about us. They frustrate us by asking for information we don’t have or presenting a set of options, none of which seem right.
But they do have one thing going for them: they are good at collecting data at large scale. In the world of impact investment, forms are proliferating as we seek to collect more data on social impact from frontline organisations. We are aware that our industry will face a crisis of credibility if we cannot demonstrate that our investments are having a social impact, and asking organisations to fill in forms about their impact seems like a good place to start.
Outside the box
But more is not always better. The problem is that data on social impact often doesn’t fit into neat pre-defined boxes. The myriad of organisations receiving money from social investors are all working with different people in different ways to try to improve their lives. To make matters worse, the quality of data collected on impact is highly variable and poorly understood. Even the most thoughtfully designed forms frustrate investees as they are asked for information that just doesn’t feel relevant. Investors are frustrated too: it is often unclear what conclusions can be drawn from all of this data if we don’t know how it was collected. This data isn’t helping anyone to make decisions.
This crisis of credibility leaves social investors focusing on the easy metrics like numbers reached or the number of loans approved, leaving a real examination of whether lives have improved to one side.
It is one of the great joys of my job that I don’t have to subject our investees to anonymous forms. As a venture capital investor, our model is one of high engagement and our portfolio is small. I have the luxury of time to be responsive and structure metrics around what each organisation needs. We work together to find an approach to data collection that both tells us what impact is being achieved and also helps the investee manage its business.
The resulting mix of metrics and approaches requires judgement when communicating the impact of our fund to investors. The important questions are difficult to answer. But that’s OK. That’s my job and I am resourced to do it.
Bridging the gap
When I listen to the challenges faced by my colleagues in social investors that work at 10 times or 100 times the scale that we do, I realise what a gulf there is between these two worlds. They simply do not have the time to apply what I would regard as best practice approaches to impact measurement to every grant, loan or investment in their portfolio.
It isn’t helpful for me to say that good impact measurement requires sitting down with your investee, talking through the theory of change and reviewing the existing evidence on best practice. They know that. There just isn’t the resource in either the investors or the investees to do it.
Intermediaries like us sit in the middle of competing desires, on the one hand there are the investees who want any reporting around impact to be tailor-made to their needs; on the other there are the asset owners who, understandably, want a compelling account of the impact their money is having in aggregate. If we can’t square this circle and convince asset owners that we really make a difference, and social-purpose organisations that they should want our money, we will cease to exist.
How we improve our account of impact is an important conversation, and intermediaries of all kinds are getting better at sharing their approaches. But the problem of improvement at scale is not simply a case of learning from best practice. Especially when best practice is most commonly found at small scale. How do we scale up a deep understanding of what achieves impact when budgets and time are constrained?
I don’t know what the answer is, maybe some hybrid approach where large-scale operators randomly select a proportion of investments for more in-depth evaluation. Or as automated data collection and processing becomes increasingly widespread, technology may be part of the solution.
One thing I do know is that we won’t find the answer unless people like me, with the luxury of small portfolios, recognise that the answer is not simply to scale up our high-intensity approach, and people at the other end of the spectrum recognise the urgency with which we need to demonstrate impact in a more credible way that justifies its existence as a separate asset class.
Standardised forms are not the (whole) answer, but what is?
– See more at: http://www.nesta.org.uk/blog/measuring-impact-investments-question-scale#sthash.dPfuuF4M.dpuf