Evaluating Online Safety Education: A Framework

Working with children to understand what it takes to create an effective online safety resource.

Our Vision

Cyberbullying and online grooming are major problems impacting the safety of children online. Many risk reduction programmes have been designed and implemented in response, seeking to empower victims and change perpetrator behaviour. As they move from design to content development and delivery, these programmes involve large investments of time and money. Yet, are they effective? Do they lead to significant and lasting behaviour change? Few programmes have been evaluated, meaning this question could not be answered. We're working to address that gap.

Our Project Plan

We teamed up with UNICEF EAPRO consultants to research cyberbullying and online grooming indicators, and develop an indicator framework to understand if interventions result in behaviour change for child online protection.

To make sure the framework included the current “state-of-the-art” approach to online safety evaluations, we carried out a rapid literature review of online safety projects conducted by scholars, NGOs and other non-profits. This review helped us to understand the different approaches to online safety evaluations, including their strengths and weaknesses. This review also looked at the behavioural change theories that support these interventions. Based on this literature review, our team developed its own theory of change. Our composite model draws from multiple theories and can be adjusted to fit different contexts.

Using reviews of both evaluation approaches and change theories, we put together a database of indicators. From self-reported questionnaires to platform reports and participant stories, these indicators provide a range of options to understanding the effectiveness of online safety programmes.

We greatly value the experiences of those on the front lines of online safety. To tap into that expertise we  consulted with UNICEF’s Think Tank, a global panel of online safety experts throughout the project. To gain an industry perspective on this material, the team also interviewed technology providers, government agencies, and child-focused companies, including Facebook, Microsoft, eSafety Australia, Lego, Roblox, and a Cambodia-based NGO, Action Pour Les Enfants (APLE). Finally, to make sure that children’s voices were heard and reflected in the framework, the team conducted two workshops with children in Cambodia, facilitated by APLE.

These suggestions helped create a short list of indicators and theories of change, and informed the overall framework. Detailed in the final report, the framework itself uses a range of scales and types of programmes that focus on different aspects and stages of behaviour change: raising awareness, reporting harmful activity and empowering young people themselves to support each other through conversations and actions on cyberbullying and grooming.

What Impact will this research have?

Few online safety education initiatives are informed by current evidence, and even less are evaluated for their impacts on children's behaviour change. To change this, the UNICEF EAPRO project focuses on measuring children’s behaviour change, so they can support governments, NGOs and the technology sector to assess the impact of online safety interventions.

The project findings and evaluation tool are a freely available public good, and this framework has the capability to scale from large interventions by global tech providers to smaller campaigns administered by non-government organisations (NGOs) and civil society organisations (CSOs). In each of these cases, the framework should help organisations understand the scale and quality of impact, and help them to make sure their messages of online safety are aligned with an evidence-based theory of change.

Outputs

Online Safety Evaluation Framework Generator
To make this framework easy to use, the team developed an online prototype tool – Evaluate It! Our tool guides you to choose the topic, theory of change, and indicators that are relevant to your organisation through a drag-and-drop interface. Once you have completed the process, a dynamic PDF document containing your custom framework is generated, alongside some key background information on evaluation and suggestions for best-practice use.

The theory of change, indicators, and online tool form an accessible and cohesive evaluation framework, improving online safety evaluation by providing a more holistic and evidence-based means of measurement. While this framework is intended for technology sector companies, it can also be used by NGOs and CSOs to better understand programme impact, and ensure messages are aligned with an evidence-based theory of change.

It is planned that this prototype will be developed into a usable and user friendly online tool with accompanying guidance in Phase 2 of the project.

UNICEF's  Let's Chat About Online Grooming Campaign
Our friends at UNICEF have also created the Let's Chat About Online Grooming Campaign to combat online grooming and sexual exploitation in Cambodia, to the Internet safer for children.

If you're a young person, you may like to find out more about how you can stay safe online. There are also tips for parents and caregivers to keep children safe online.

Evaluating Online Safety Initiatives: How to build the evidence base on what works to keep children safe online
Read the full project report here. 

Collaboration Team

We are grateful to the wide and varied skill sets our collaborators brought to the table, making this idea a reality.

Internal collaborators
Professor Amanda Third
Associate Professor Liam Magee
Professor Heather Horst
Associate Professor Ann Dadich
Dr Luke Munn
Ms Vanicka Arora
Ms Lilly Moody