A Multi-Site Trial of Mentoring: supporting small, grassroots, organisations to participate in robust impact evaluations

YMCA George Williams College has partnered with the Centre for Evidence and Implementation and Bryson Purdon Social Research on a ground-breaking multi-site randomised controlled trial (RCT) of mentoring for young people. This research – the first of its kind - is funded by the Youth Endowment Fund (YEF) and is testing a different approach to running an RCT, drawing together a cohort of smaller organisations to deliver mentoring using a shared model of practice. Together, these smaller grassroots organisations are collectively working with a group of young people large enough to be part of an RCT; an opportunity that is usually out of reach to these organisation. This multi-site RCT is evaluating the impact of mentoring on young people aged 10-14 who are at increased risk of involvement in serious youth violence.

As with many RCTs, the research started with a feasibility phase, to assess whether carrying out research in this way was feasible and ‘acceptable’ to the youth organisations and young people taking part.

We started small, with nine Delivery Partner Organisations (DPOs) recruiting a total of 93 young people. The DPOs were involved in co-designing a ‘shared practice model’ for mentoring and all delivered their programmes adhering to this model. This included working with paid mentors rather than volunteers, and facilitating weekly mentoring sessions for at least 12 weeks. Young people referred for mentoring were ‘randomised’ into either a control group or an experimental group. Young people in the control group waited 12 weeks from the point of referral before starting mentoring, whilst those in the experimental group started mentoring straight away. Young people in the control group were offered alternative group-based activities whilst they waited for their mentoring to start. All young people completed baseline and 12-week follow-up surveys, so that we could look at the differences in outcomes for the two groups – which we hope will show the potential impact of mentoring.

Promising results from the initial phase, in terms of feasibility and acceptability, inspired a move on to the efficacy phase. The efficacy trial, which is still underway until the end of this year, is focused specifically on the impact of mentoring on young people. The efficacy phase aims to work with 850 young people across seventeen DPOs, fifty young people per organisation.

Multi-site trials are exciting as they provide an opportunity for smaller, grassroots organisations to generate robust evidence of the impact of a common intervention like youth mentoring by aggregating results across multiple organisations. However, whilst taking a collective approach reduces the barrier to engagement for smaller organisations, involvement in an RCT is still resource-intensive, and can introduce particular constraints and requirements into informal and non-formal provision for young people. It’s critical that the research team takes the time to learn about and understand all the delivery organisations involved. Without understanding the individual contexts in which organisations are operating, it is impossible to provide the support that grassroots organisations often need to meet the requirements of the trial.


Supporting the Delivery Partner Organisations

The research team ensured that there were diverse support options for DPOs, such as online information sheets and a peer support network. However, we found that one-to-one support was the most useful for organisations within the trial. DPOs were working in different contexts with varying levels of complexity, and often what was needed was bespoke advice and a space to share problems and air tensions as part of a consistent and trusted relationship. Taking the time to speak directly with project delivery staff and learn about their local contexts, referral mechanisms, and delivery settings was an essential component of the research. Any future trial of this nature must carve out sufficient time to build relationships with and knowledge about delivery organisations and the communities in which they work. Not doing this will likely pose risks to the sample size and the quality of the data – and thus the trial itself. Building these kinds of relationships is important in any RCT, but the importance - and the time required – is significantly increased in a multi-site trial.

For example, DPOs working with families facing complex and multiple disadvantages often encountered parents and carers with reservations about their young person taking part. By taking time to learn more, the research team was able to establish that this fear often correlated with families’ previous experience with children’s social care and a resulting anxiety around data sharing.

We worked with the DPOs to develop strategies to rebuild trust with these families. Stressing the independence of the research, distancing ourselves from statutory agencies, and taking time to talk with families face-to-face to explain the benefits of the trial helped rebuild trust and ultimately increased the numbers of young people we were able to engage in mentoring as part the trial.  
 

Referral pathways

The shared practice model did not specify referral routes through which young people could be engaging in mentoring. This was intentional, in order to respond to the variety of contexts in which DPOs were working, and the pre-existing relationships with local partners and agencies. In many cases, DPOs were providing a range of provision for young people, and ‘referral’ was in effect signposting young people from one offer to another. However, the scale of the efficacy trial meant organisations relying on these kinds of community-based referrals struggled to engage the numbers of young people required in the timescale. Using our holistic knowledge of the DPOs processes, the research team were able to identify that DPOs utilising school-based referrals were experiencing much greater success. We were able to share this knowledge with DPOs that hadn’t worked with schools before. We encouraged relationship-building with schools and suggested awareness-raising approaches such as hosting school assemblies and parent/carer evenings to promote their offer of mentoring as part of the trial.

Involvement in an RCT is always likely to involve some requirement to flex or adapt ways of working. With a multi-site trial, the support needed to work alongside and guide multiple organisations through this process is considerable. The process is aided by setting clear expectations with delivery organisations from the outset. Organisations must be willing to flex their ways of working, build new relationships and adapt to new settings to maximise the possibility of fulfilling trial requirements. The research team needs to be willing and able to ‘walk with them’ on this journey.


Delivery settings

As noted earlier, many DPOs found that they needed to start working with new partners in order to engage sufficient numbers of young people in their mentoring offer as part of the trial. We found that adapting to a new delivery setting – specifically, the school environment- was something that many DPOs had to grapple with.

There have been undoubted benefits to this approach, with DPOs reporting building new and positive relationships with schools in their communities. They’ve been able to reach young people they ordinarily wouldn't and mentor young people who wouldn’t have ordinarily met their threshold for support in the community.

Being open to supporting this approach not only increased the overall reach of the trial, but also resulted in DPOs building new and valuable relationships with schools. These relationships have put DPOs in a position where they can work with school leaders to embed other services into the school environment encouraging a more relational, whole-school approach to wellbeing.

Equally, DPOs have in some cases had to manage tensions between a consistent, voluntary relationship with young people and a more formal, ‘controlled’ setting within a school. DPOs have had to negotiate the ‘place’ of mentoring within a tightly-defined school timetable, and have often found it scheduled to replace arts or sports, frequently the favourite lessons identified by young people involved in mentoring. DPOs have also had to grapple with disciplinary approaches such as exclusions and managed moves resulting in mentors turning up only to find the young person is not in school. One organisation reflected that often the reason for the exclusion or move is due to the precise issue they are working on during mentoring sessions.

In most cases, DPOs have been able to advocate for young people to have sessions at times that suit them and in the case of managed moves, where feasible, they’ve been able to travel to the young people's new school to deliver mentoring. These are encouraging and positive examples of collaboration that benefits the young people, the school and the DPO. It is also an example of the importance of the ‘bridge’ role we have played between the DPOs and the requirements of the trial. 

It’s been inspiring to be part of a support network for DPOs, learning from them, sharing knowledge, and watching as they learn and adapt to new ways of delivery. They have done everything they can to support us to build an evidence base for the impact of youth mentoring for those at risk of violence affecting young people, whilst never losing sight of what is at the heart of youth mentoring; the relationship between a young person and a trusted and emotionally available adult.

--

To read the full findings from the Multi-site Trial feasibility study, please click here.