"Just a strange pic": Evaluating 'safety' in GenAI Image safety annotation tasks from diverse annotators' perspectives
Not on the map. Why?
Abstract (arXiv)
Understanding what constitutes safety in AI-generated content is complex. While developers often rely on predefined taxonomies, real-world safety judgments also involve personal, social, and cultural perceptions of harm. This paper examines how annotators evaluate the safety of AI-generated images, focusing on the qualitative reasoning behind their judgments. Analyzing 5,372 open-ended comments, we find that annotators consistently invoke moral, emotional, and contextual reasoning that extends beyond structured safety categories. Many reflect on potential harm to others more than to themselves, grounding their judgments in lived experience, collective risk, and sociocultural awareness. Beyond individual perceptions, we also find that the structure of the task itself -- including annotation guidelines -- shapes how annotators interpret and express harm. Guidelines influence not only which images are flagged, but also the moral judgment behind the justifications. Annotators frequently cite factors such as image quality, visual distortion, and mismatches between prompt and output as contributing to perceived harm dimensions, which are often overlooked in standard evaluation frameworks. Our findings reveal that existing safety pipelines miss critical forms of reasoning that annotators bring to the task. We argue for evaluation designs that scaffold moral reflection, differentiate types of harm, and make space for subjective, context-sensitive interpretations of AI-generated content.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 37 references that are in the map. First posted on arXiv 21 Jul 2025.
Builds on
15 of 37 references.
References · 15
- 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
- 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
- 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
- 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
- 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
- 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
- 2023AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asiacited by 54
- 2020Bringing the People Back In: Contesting Benchmark Machine Learning Datasetscited by 43
- 2023How Knowledge Workers Think Generative AI Will (Not) Transform Their Industriescited by 30
- 2025Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methodscited by 15
- 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
Built on by
Not cited yet.