Using Our Measures to Promote Quality Improvement in Evaluation | Our Thoughts – Steve Hillman

For this month's Our Thoughts, Head of Partnerships Steve Hillman reflects on how the Centre's measurement tools are helping to champion quality improvement in evaluation.
 

Over the past year, the Centre for Youth Impact has worked with its research partner QTurn to develop and launch a set of measurement tools allied to our updated Outcomes Framework 2.1. These tools and resources are all designed to enable youth organisations to better understand the relationship between the quality of their provision and outcomes for young people in their provision. The Centre has been using these measurement tools with youth organisations and young people in two evaluations we have been conducting throughout 2021 and 2022, the Premier League Kicks Targeted programme and the YMCA England and Wales Y’s Girls mentoring programme.

Premier League Kicks Targeted, part of the wider Kicks programme, is a Premier League Charitable Fund and BBC Children in Need jointly funded community programme, which seeks to reach young people who are at risk of violence through reducing risk factors and increasing protective factors, with the latter closely aligned to the importance of socio-emotional learning skills, as evidenced in Framework 2.1. Y's Girls was developed by YMCA Scotland and has been implemented and shaped by YMCA England & Wales , YMCA Scotland and YMCA Ireland, which supports young women and girls aged 9-14 who are experiencing challenges, such as poor self-esteem and lack of positive role models, to build resilience and confidence in an informal environment.

Using our measurement tools has helped to generate insight into the quality of staff interaction with young people, the growth and development of socio-emotional skills of young people participating, and the level of ‘mental engagement’ that young people are experiencing with the provision on offer.

In the evaluation of Premier League Kicks Targeted, we are using the following tools:

• The Staff Programme Quality Survey (SPQS) - a 14 question survey that asks staff about their practices in engaging with young people and how frequently they use a series of practices that can be demonstrated to improve young people’s socio-emotional skills development; and
• The Youth Engagement Survey (YES) - a 10 question survey that asks young people about their levels of mental engagement with provision (for example, whether they felt heard and respected, able to contribute and appropriately challenged by activities).
   

Each of these tools are being used to gather data at three points of time over the life of the programme.

For the Y’s Girls evaluation, we are using the SPQS with mentors and the YES, alongside the following tools:
 

• The Youth Report of Skills Survey (YRSS) - a 26-question survey that asks young people to reflect on their socio-emotional skills tied to the six outcome domains on our Outcomes Framework (for example, empathy and problem solving); and
• The Adult Rating of Youth Behaviour (ARYB) - the parents and carers of the young women on the programme complete the ARYB at the start and end of the programme. The ARYB is an observation-based tool that asks adults to observe young people’s behaviours and rate them by frequency according to a series of questions, again tied to the domains of our Outcomes Framework.

Though both evaluations are still ongoing, the data we have generated so far has enabled us to draw out initial insights. Overall, the level of young people’s mental engagement with provision in both programmes is high. This is encouraging because strong mental engagement creates the right conditions for young people to enhance their socio-emotional skills. At the same time, data shows the quality of practice broadly follows the pattern of data we have gathered in other evaluations. Mapping data to the Quality Pyramid (below) that underpins our work on continuous quality improvement, the highest scores attained in both programmes were in the creation of safe environments for young people – the ‘foundation’ of quality provision. This was followed by supportive environments.  In the case of Y’s Girls, the scores model our existing data closely, with interactive environments coming next and engaging environments last. This is not surprising, as each ‘domain’ of the pyramid builds on the one below, with young people being supported to reach higher levels of engagement and leadership over time. However, in the case of Premier League Kicks Targeted, the engaging environments domain scores third and interactive last. This is interesting, as it suggests that specific elements of the programme are directly supporting high level engagement. We believe this is connected to the structured goal-setting and progress monitoring with young people that is a key feature of the Premier League Kicks Targeted programme and is a significant part of the engaging environments domain on the SPQS.
 

For Premier League Kicks Targeted, we have facilitated a series of capacity-building webinars for each of the four quality pyramid domains, and were able to focus the content of those webinars using their scores on the SPQS.  For example, in the ‘Interactive Environments’ webinar, we explored how to incorporate skills associated with teamworking or working in a group in a programme where much of the provision is offered one to one.

With Y’s Girls, drawing on YRSS and ARYB data, we have been able to identify the outcome domains where girls and young women have greatest room for growth, and where attention should be focused in the mentoring programme.

As these evaluations draw to a close over the coming months, we hope to generate further insight into how far the quality of the programme has increased over time, the role of measurement and support in improving programme quality, the relationship between quality and outcomes, and where attention should be focused in future rounds of evaluation.

In 2022, we revised our Outcomes Framework and associated measures to further reflect the importance of socio-emotional skill development. We have also developed a resource hub to guide you through the process of programme design, quality delivery, measurement, impact evaluation and organisational learning, which may be of interest to you if you wish to embark on a journey similar to YMCA England and Wales and Premier League.

To find out more about our measurement tools and how you can get involved, click here.