Impact measurement: Why, exactly?

In an article originally published in Campbell Tickell’s CT Brief, the Centre’s Pippa Knott explores charities’ different objectives in undertaking impact measurement – and on whose behalf. She argues that the mission must be to understand impact and care about getting meaningful answers, rather than seeking tick-box ways to respond to others’ demands.

The arguments for why charities should measure the impact of their work are well rehearsed. Some say that charities must measure impact to be accountable to existing funders, or able to market their work to new ones – both pragmatic responses to external context. Others set out ethical responsibilities to understand and improve interventions (and stop anything that isn’t working), or to support beneficiaries to reflect on their own progress. Others again are interested in increasing sector-wide knowledge about society and social programmes.

But are these objectives mutually reinforcing? Can they all be met by the same impact measurement activities, or by a single organisational function? Might there be contradictions between some of them? And more fundamentally, which relate most directly to sustainability for the most effective socially focused work?

Impact measurement has been one of the most dominant – and disputed – features of the evolution of the charitable sector in recent decades. Its profile has risen in line with debates about new public management, social value and austerity. Reasonable consensus exists as to what impact measurement is attempting to achieve. The Big Lottery Fund defines impact measurement as “the process of trying to find out what effect an intervention is having on people, organisations or their external … environment”. Measurement practice often seeks evidence of positive impact on outcomes: the changes brought about in the lives of beneficiaries. Activities associated with impact measurement include: producing narratives of how impact is intended to be achieved (logic models or theories of change); gathering monitoring or outcomes data, often from beneficiaries via self-report surveys; or gathering stories of individual lives transformed.

Exactly how, why and for whom impact measurement “gets done” is more confused and contentious. Many people will have views on what “evidence” a charity should collect, and how. These people may be internal (managers, fundraisers, practitioners, beneficiaries) or external (policy makers, funders, consultants, academics). Varying pressures on social organisations to measure the impact of their work have resulted in a flurry of data collection, and strong debate about what constitutes evidence, both “good” and “bad”. But there is a risk that this data is more about responding to immediate pressures, and misses the point of the relational work that is at the heart of socially orientated work.

To be valuable in the widest sense, impact measurement needs to be done with curiosity and honesty: a desire to use evaluation to become more effective, rather than to reinforce existing views on effectiveness. It also needs to be a collective endeavour, rather than an exercise in competitive advantage for individual organisations. However, the environment in which charities operate is rarely conducive to this, and the incentives are weak. For many, meaningful impact measurement requires a difficult process of culture change, and a mind-set shift for policy makers and funders as much as for delivery organisations.

The Centre for Youth Impact was established in 2014 to support organisations that work with young people to change their practice in relation to impact measurement. In partnership with our networks we are developing approaches that are valuable to the statutory and voluntary organisations that make up today’s youth sector with the aim of improving provision and moving it to a more sustainable footing.

In the midst of the complexities and tensions around impact measurement, charities need to establish organisation-wide clarity about why (and for whom) they are measuring, and whether they are developing insight that will enable practice – and outcomes for young people – to improve as a result. They need to be able to access tools that will enable them to do this, and align their practice with their peers. We believe that the key questions that every organisation should be asking with openness as to what they might learn and do differently as a result are: Why do you do what you do? What exactly are you doing? Are you doing it consistently well? Are you true to your premises? What do your beneficiaries think of what you do? Are you achieving your aims?
​
When individuals at all levels understand the questions at the heart of impact measurement and care about getting meaningful answers, the resultant data is more likely to reflect the realities of practice and be used to improve it. The mission must be to understand impact, rather than seeking tick-box ways to respond to others’ demands. There is a technical aspect to this, but relationships and culture matter much more.