Why proof is not enough

This blog was written by The Centre’s first Chair, Thomas Lawson, who stood down last month in order to focus on his new role as CEO at Turn2Us, part of the critical fight against poverty in the UK today. In it, Tom shares his reflections on the need for a culture of practice that focuses on learning how to make continual improvements to our work.

The Centre for Youth Impact, as an independent charity, is only just approaching its first birthday, but the project is now five years old.  Its origins lie in the shift of the youth policy brief to the Cabinet Office, and the growing interest in the evidence and impact agenda. In 2014, the Centre was brought to life by Susanne Rauprich of NCVYS, Stephen Bediako of The Social Innovation Partnership and Project Oracle, and Bethia McNeil, then at Dartington Social Research Unit and now the Centre’s Chief Executive. The aspiration seemed clear: Cabinet Office wanted better proof of the value of youth work and the informal and non-formal youth provision offered by a range of bodies for young people.  If that provision was going to be funded by government, it needed a strong argument to be able to secure the income our organisations needed to support young people to thrive and lead. If that evidence could be generated and drawn together, then those programmes could be ’scaled’, rolled out across the country or, if they already existed, simply funded.
 
It remains the case that many view the best tool in town to secure that evidence as the ‘gold standard’ randomised control trial (RCT).  The argument is that there is no better tool for ‘proving’ whether an intervention has a positive impact on pre-determined outcomes.  What is clear is that an RCT is best-suited to a programme that has been tightly designed for an environment in which the variables will be reasonably constant, where need is well understood, where intended outcomes are clearly defined in advance, where context will be reasonably similar in different regions and environments across the country, and where change can be measured both at the end of a programme and again some time afterwards.
 
Picture this.  An organisation spends several months, working with young people, gathering evidence and designing a concept for a programme. It believes strongly in the potential of this programme to affect change in the lives of young people, and feels that an RCT would provide the evidence it has been looking for to prove its impact. The organisation applies for funding for the programme, alongside working with academics to design and secure funding for an RCT. The process takes nine months to secure the income.  In the programme's first year, the organisation is recruiting staff, finalising partners, refining the design and protocols for the RCT and then beginning the work; some things won’t work out like the design had hoped and there will be a couple of false starts in areas of the programme.  By the end of the second year, all things being well, the team is gelling, and young people are engaged.  By the end of the third year, the programme has really begun to motor - probably four years after its original design.  However, since then, new technologies are in play, new social dynamics have emerged (think about the steep rise in serious youth violence), funding of partner organisations (on which the programme relied) has shrunk, local politics have changed - all of which has a profound impact on the lives of the young people for whom the programme was designed, and the context for the delivery itself. The RCT is probably going to show the programme does not work, likely as not because the environment was too volatile, and the research design didn’t take account of the need to adapt - but crucially, it is also nearly impossible for it to suggest what would have worked instead.  The RCT, whilst a gold standard measurement tool, may not be what our sector needs.
 
What we need is a culture across all of our practice that makes us hunger to learn how to continually improve our work, rather than searching for the silver bullet of proving it for the sake of funding and favour.
 
Over the last couple of years, the Centre for Youth Impact has been working hard to try to support the growth of that culture - in partnership with hundreds of people and organisations.  In particular it has been trying to drive forwards more understanding about collective impact - an approach that flexes with the reality of the situation for young people, co-designed with them and delivered in multi-lateral, cross-sector partnerships. Collective impact favours a deep understanding of the real-time qualitative and quantitative insight that suggests where we need to reflect, listen and adapt.
 
There is a chance, in partnership and in the service of young people and their talent today, and in their futures, that our wonderful youth sector could begin to show what continuous improvement looks like.  I look forward to closely following the Centre’s work this year and beyond, with a continued focus on questioning, learning and creating change.
 
Serving the Centre for Youth Impact as its chair has been a great privilege.  I wish the Centre’s staff, trustees, partners the very best for its work.