mlsanityData to 7 Oct 2026

Annotation alignment: Comparing LLM and human annotations of conversational safety

arXiv 2406.06369 ↗First posted 10 Jun 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Do LLMs align with human perceptions of safety? We study this question via annotation alignment, the extent to which LLMs and humans agree when annotating the safety of user-chatbot conversations. We leverage the recent DICES dataset (Aroyo et al., 2023), in which 350 conversations are each rated for safety by 112 annotators spanning 10 race-gender groups. GPT-4 achieves a Pearson correlation of r = 0.59 with the average annotator rating, textithigher than the median annotator's correlation with the average (r=0.51). We show that larger datasets are needed to resolve whether LLMs exhibit disparities in how well they correlate with different demographic groups. Also, there is substantial idiosyncratic variation in correlation within groups, suggesting that race & gender do not fully capture differences in alignment. Finally, we find that GPT-4 cannot predict when one demographic group finds a conversation more unsafe than another.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

20 of 37 references.

References · 20

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023Whose Opinions Do Language Models Reflect?cited by 759
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  7. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  8. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  9. 2020With Little Power Comes Great Responsibilitycited by 138
  10. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  11. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  12. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
8 more
  1. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  2. 2021Designing Toxic Content Classification for a Diversity of Perspectivescited by 63
  3. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  4. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  5. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  6. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguardscited by 17
  7. 2023GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectivescited by 16
  8. 2023Coarse race data conceals disparities in clinical risk score performancecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  2. 2025When Large Language Models are Reliable for Judging Empathic Communicationcited by 21
  3. 2025Towards Automating Data Access Permissions in AI Agentscited by 20
  4. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  5. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behaviorcited by 8
  6. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensivenesscited by 7
  7. 2025WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thaicited by 3
  8. 2025Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Casescited by 2
  9. 2025Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMscited by 2
  10. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  11. 2025Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Modelscited by 1
  12. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
10 more
  1. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysiscited by 1
  2. 2026From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?cited by 1
  3. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  4. 2025Are Humans as Brittle as Large Language Models?cited by 0
  5. 2025Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scalecited by 0
  6. 2026Understanding Annotator Safety Policy with Interpretabilitycited by 0
  7. 2026Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requestscited by 0
  8. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluationcited by 0
  9. 2026Aligned in Form, Not in Meaning: The Comprehension - Containment Decoupling of LLM Safety in Low-Resource Bangla Derogatory Speechcited by 0
  10. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
  3. 2026Dataset Signatures in Human-LLM Interactions and User Modeling
  4. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  5. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
  6. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
  7. 2024Impact of Decoding Methods on Human Alignment of Conversational LLMs
  8. 2025Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach