Evaluating Online Safety Education: A Framework

Working with children to understand what it takes to create an effective online safety resource

Children playing with tablet

Our Vision

Cyberbullying and online grooming are major problems impacting the safety of children online. Many risk reduction programmes have been designed and implemented in response, seeking to empower victims and change perpetrator behaviour. As they move from design to content development and delivery, these programmes involve large investments of time and money. Yet, are they effective? Do they lead to significant and lasting behaviour change? Few programmes have been evaluated, meaning this question could not be answered. We're working to address that gap.

Our Project Plan

We teamed up with UNICEF EAPRO consultants to research cyberbullying and online grooming indicators, and develop an indicator framework to understand if interventions result in behaviour change for child online protection.

To make sure the framework included the current “state-of-the-art” approach to online safety evaluations, we carried out a rapid literature review of online safety projects conducted by scholars, NGOs and other non-profits. This review helped us to understand the different approaches to online safety evaluations, including their strengths and weaknesses. This review also looked at the behavioural change theories that support these interventions. Based on this literature review, our team developed its own theory of change. Our composite model draws from multiple theories and can be adjusted to fit different contexts.

Using reviews of both evaluation approaches and change theories, we put together a database of indicators. From self-reported questionnaires to platform reports and participant stories, these indicators provide a range of options to understanding the effectiveness of online safety programmes.

We greatly value the experiences of those on the front lines of online safety. To tap into that expertise we  consulted with UNICEF’s Think Tank, a global panel of online safety experts throughout the project. To gain an industry perspective on this material, the team also interviewed technology providers, government agencies, and child-focused companies, including Facebook, Microsoft, eSafety Australia, Lego, Roblox, and a Cambodia-based NGO, Action Pour Les Enfants (APLE). Finally, to make sure that children’s voices were heard and reflected in the framework, the team conducted two workshops with children in Cambodia, facilitated by APLE.

These suggestions helped create a short list of indicators and theories of change, and informed the overall framework. Detailed in the final report, the framework itself uses a range of scales and types of programmes that focus on different aspects and stages of behaviour change: raising awareness, reporting harmful activity and empowering young people themselves to support each other through conversations and actions on cyberbullying and grooming.

What Impact will this research have?

Few online safety education initiatives are informed by current evidence, and even less are evaluated for their impacts on children's behaviour change. To change this, the UNICEF EAPRO project focuses on measuring children’s behaviour change, so they can support governments, NGOs and the technology sector to assess the impact of online safety interventions.

The project findings and evaluation tool are a freely available public good, and this framework has the capability to scale from large interventions by global tech providers to smaller campaigns administered by non-government organisations (NGOs) and civil society organisations (CSOs). In each of these cases, the framework should help organisations understand the scale and quality of impact, and help them to make sure their messages of online safety are aligned with an evidence-based theory of change.

Outputs

Online Safety Evaluation Framework Generator
To make this framework easy to use, the team developed an online prototype tool – Evaluate It! Our tool guides you to choose the topic, theory of change, and indicators that are relevant to your organisation through a drag-and-drop interface. Once you have completed the process, a dynamic PDF document containing your custom framework is generated, alongside some key background information on evaluation and suggestions for best-practice use.

The theory of change, indicators, and online tool form an accessible and cohesive evaluation framework, improving online safety evaluation by providing a more holistic and evidence-based means of measurement. While this framework is intended for technology sector companies, it can also be used by NGOs and CSOs to better understand programme impact, and ensure messages are aligned with an evidence-based theory of change.

It is planned that this prototype will be developed into a usable and user friendly online tool with accompanying guidance in Phase 2 of the project.

UNICEF's  Let's Chat About Online Grooming Campaign
Our friends at UNICEF have also created the Let's Chat About Online Grooming Campaign to combat online grooming and sexual exploitation in Cambodia, to the Internet safer for children.

If you're a young person, you may like to find out more about how you can stay safe online. There are also tips for parents and caregivers to keep children safe online.

Evaluating Online Safety Initiatives: How to build the evidence base on what works to keep children safe online
Read the full project report here. 

Collaboration Team

We are grateful to the wide and varied skill sets our collaborators brought to the table, making this idea a reality.

Internal collaborators
Professor Amanda Third
Associate Professor Liam Magee
Professor Heather Horst
Associate Professor Ann Dadich
Dr Luke Munn
Ms Vanicka Arora
Ms Lilly Moody

External collaborators
The Y&R project team collaborated closely with Emma Day (Consultant); Keith Woo (Consultant), and Rachel Harvey (Regional Adviser Child Protection) from UNICEF East Asia and Pacific Regional Office.

Project support
A Think Tank of international experts in child protection provided technical guidance throughout the process: Elizabeth Milovidov – independent lawyer and expert on digital parenting; Priyanka Bhalla – Director of Social Impact, Quilt.Ai; Thanh Bui Duy – Online Regional Safety Specialist, Child Fund Vietnam; Patrick Burton – Executive Director, Center for Justice and Crime Prevention; Professor David Finkelhor – University of New Hampshire; Professor Hany Farid, School of Information, UC Berkeley; Carla Licciardello, Child Online Protection Focal Point, ITU; Alexandru Caciuloi, Regional Programme Coordinator Cybercrime and Cryptocurrencies, UNODC; Gabrielle Berman – Senior Adviser, Ethics, UNICEF; Karuna Nain, Global Safety Policy Lead – Facebook; Chelsey Le Page – Social Impact Partnerships & Programs Lead, Facebook; Julia Fossi – Director, Office of e-Safety Commissioner; Daniel Kardefelt-Winter – UNICEF Innocenti; Rudrajit Das – Chief, Communication for Development, UNICEF EAPRO; Benjamin Grubb, Business Analyst, Technology for Development, UNICEF EAPRO; Anjan Bose, Child Online Protection Specialist, UNICEF; Dr. Monica Bulger – Digital Literacy Specialist; Marie-Laure Lemineur, Deputy Executive Director, ECPAT International; Manisha Dogra – VP Sustainability Asia, Telenor; Liz Thomas – Regional Digital Safety Lead, Asia Pacific, Microsoft; and Ysrael Diloy, Senior Advocacy Officer, Stairway Foundation Philippines.

Funding

This project was commissioned by the UNICEF East Asia and the Pacific Regional Office.

Unicef Logo

Research Streams

Places and Platforms
Participation and Engagement

Period

February 2020 – April 2022

Contact

If you would like to get in contact with the  Evaluating Online Safety Education team, please email Professor Amanda Third at a.third@westernsydney.edu.au