# CAREBench: A Child-Safety Risk Benchmark for Language Models

arXiv: https://arxiv.org/abs/2606.29685 · HTML: https://mlsanity.com/paper/2606.29685/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 24 of 24)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types](https://mlsanity.com/paper/2410.21965/) (2024)
- [The opportunities and risks of large language models in mental health](https://mlsanity.com/paper/2403.14814/) (2024)
- [Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks](https://mlsanity.com/paper/2310.07879/) (2023)
- [Generating Phishing Attacks using ChatGPT](https://mlsanity.com/paper/2305.05133/) (2023)
- [Emotional Manipulation by AI Companions](https://mlsanity.com/paper/2508.19258/) (2025)
- [Next-Generation Phishing: How LLM Agents Empower Cyber Attackers](https://mlsanity.com/paper/2411.13874/) (2024)
- [People are poorly equipped to detect AI-powered voice clones](https://mlsanity.com/paper/2410.03791/) (2024)
- [MinorBench: A hand-built benchmark for content-based risks for children](https://mlsanity.com/paper/2503.10242/) (2025)
- [Understanding Generative AI Risks for Youth: A Taxonomy Based on Empirical Data](https://mlsanity.com/paper/2502.16383/) (2025)
- [Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions](https://mlsanity.com/paper/2506.13510/) (2025)
- [Exploring Parent-Child Perceptions on Safety in Generative AI: Concerns, Mitigation Strategies, and Design Implications](https://mlsanity.com/paper/2406.10461/) (2024)
- [LLM Safety for Children](https://mlsanity.com/paper/2502.12552/) (2025)
- [SproutBench: A Benchmark for Safe and Ethical Large Language Models for Youth](https://mlsanity.com/paper/2508.11009/) (2025)
- [Unveiling AI's Threats to Child Protection: Regulatory efforts to Criminalize AI-Generated CSAM and Emerging Children's Rights Violations](https://mlsanity.com/paper/2503.00433/) (2025)
- [YouthSafe: A Youth-Centric Safety Benchmark and Safeguard Model for Large Language Models](https://mlsanity.com/paper/2509.08997/) (2025)
- [Exploring Families' Use and Mediation of Generative AI: A Multi-User Perspective](https://mlsanity.com/paper/2504.09004/) (2025)

## Built on by (most cited first, 1 of 1)

- [CompanionSafe: From Multi-turn Safety Evaluation to Guardrails for AI Companions](https://mlsanity.com/paper/2610.18095/) (2026)

## Abstract (arXiv)

How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm? Existing child safety evaluations focus on child sexual abuse material, yet many child-safety failures begin earlier: in model assistance that helps adults manipulate, impersonate, profile, or isolate minors, and in model responses that deepen children's emotional dependence on AI systems rather than redirecting them toward human support. We introduce CAREBench (Child AI Risk Evaluation), a benchmark to assess such upstream child-safety risks in language models. CAREBench contains 500 prompts spanning twelve risk categories, including grooming and relationship engineering, deception and impersonation, surveillance and privacy, sextortion and sexual abuse, AI anthropomorphization, emotional dependency, and mental illness sensitivity. Developed with response annotations from parents and clinicians, the benchmark excludes explicit abuse material and imagery; instead, it evaluates whether models recognize, refuse, de-escalate, or redirect risky interactions before harm becomes overt. Evaluating seven frontier models on our benchmark, we find failure rates ranging from 2% to 58%, with failure patterns that vary across risk categories. CAREBench provides a responsibly scoped evaluation for LLM developers to identify and close gaps in child safety policies.