# Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountability

arXiv: https://arxiv.org/abs/2504.14446 · HTML: https://mlsanity.com/paper/2504.14446/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 20 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [On the De-duplication of LAION-2B](https://mlsanity.com/paper/2303.12733/) (2023)
- [The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models](https://mlsanity.com/paper/2405.04623/) (2024)
- [From colouring-in to pointillism: revisiting semantic segmentation supervision](https://mlsanity.com/paper/2210.14142/) (2022)
- [Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Models](https://mlsanity.com/paper/2311.17394/) (2023)
- [Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework](https://mlsanity.com/paper/2405.02703/) (2024)
- [How Spammers and Scammers Leverage AI-Generated Images on Facebook for Audience Growth](https://mlsanity.com/paper/2403.12838/) (2024)
- [Seeing without Looking: Analysis Pipeline for Child Sexual Abuse Datasets](https://mlsanity.com/paper/2204.14110/) (2022)
- [#PraCegoVer: A Large Dataset for Image Captioning in Portuguese](https://mlsanity.com/paper/2103.11474/) (2021)
- [CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languages](https://mlsanity.com/paper/2310.13683/) (2023)
- [Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations](https://mlsanity.com/paper/2410.23432/) (2024)

## Built on by (most cited first, 6 of 6)

- [AutiHero: Engaging Parents in Creating Personalized, Multi-path Social Narratives for Autistic Children](https://mlsanity.com/paper/2509.17608/) (2025)
- [Privacy in Image Datasets: A Case Study on Pregnancy Ultrasounds](https://mlsanity.com/paper/2602.07149/) (2026)
- [A Privacy by Design Framework for Large Language Model-Based Applications for Children](https://mlsanity.com/paper/2602.17418/) (2026)
- [Toward a More Ethical Facial Age Estimation: A Generalized Zero-Shot Benchmark Without Training on Children's Data](https://mlsanity.com/paper/2605.29230/) (2026)
- [Classifying by Proxy: Explainable and Reproducible Ensemble of Proxy Tasks for Child Sexual Abuse Imagery Classification](https://mlsanity.com/paper/2606.15993/) (2026)
- [Detecting CSAM Text-to-Image LoRAs From Weights](https://mlsanity.com/paper/2607.25750/) (2026)

## Abstract (arXiv)

Including children's images in datasets has raised ethical concerns, particularly regarding privacy, consent, data protection, and accountability. These datasets, often built by scraping publicly available images from the Internet, can expose children to risks such as exploitation, profiling, and tracking. Despite the growing recognition of these issues, approaches for addressing them remain limited. We explore the ethical implications of using children's images in AI datasets and propose a pipeline to detect and remove such images. As a use case, we built the pipeline on a Vision-Language Model under the Visual Question Answering task and tested it on the #PraCegoVer dataset. We also evaluate the pipeline on a subset of 100,000 images from the Open Images V7 dataset to assess its effectiveness in detecting and removing images of children. The pipeline serves as a baseline for future research, providing a starting point for more comprehensive tools and methodologies. While we leverage existing models trained on potentially problematic data, our goal is to expose and address this issue. We do not advocate for training or deploying such models, but instead call for urgent community reflection and action to protect children's rights. Ultimately, we aim to encourage the research community to exercise - more than an additional - care in creating new datasets and to inspire the development of tools to protect the fundamental rights of vulnerable groups, particularly children.