# DICES Dataset: Diversity in Conversational AI Evaluation for Safety

arXiv: https://arxiv.org/abs/2306.11247 · HTML: https://mlsanity.com/paper/2306.11247/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2023.

## Built on by (most cited first, 30 of 86)

- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models](https://mlsanity.com/paper/2404.16019/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions](https://mlsanity.com/paper/2504.15236/) (2025)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge](https://mlsanity.com/paper/2410.03775/) (2024)
- [Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks](https://mlsanity.com/paper/2311.09743/) (2023)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation](https://mlsanity.com/paper/2406.11852/) (2024)
- [How Far Can We Extract Diverse Perspectives from Large Language Models?](https://mlsanity.com/paper/2311.09799/) (2023)
- [Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)](https://mlsanity.com/paper/2501.19047/) (2025)
- [Holistic Safety and Responsibility Evaluations of Advanced AI Models](https://mlsanity.com/paper/2404.14068/) (2024)
- [Annotator-Centric Active Learning for Subjective NLP Tasks](https://mlsanity.com/paper/2404.15720/) (2024)
- [Pairwise Calibrated Rewards for Pluralistic Alignment](https://mlsanity.com/paper/2506.06298/) (2025)
- [Scaling Laws Do Not Scale](https://mlsanity.com/paper/2307.03201/) (2023)
- [Near to Mid-term Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2404.17047/) (2024)
- [Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2405.08597/) (2024)
- [STAR: SocioTechnical Approach to Red Teaming Language Models](https://mlsanity.com/paper/2406.11757/) (2024)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Beyond the Binary: Capturing Diverse Preferences With Reward Regularization](https://mlsanity.com/paper/2412.03822/) (2024)
- [Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems](https://mlsanity.com/paper/2502.14019/) (2025)
- [Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models](https://mlsanity.com/paper/2507.13383/) (2025)
- [K2-Think: A Parameter-Efficient Reasoning System](https://mlsanity.com/paper/2509.07604/) (2025)
- [Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations](https://mlsanity.com/paper/2403.06009/) (2024)
- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)

## Abstract (arXiv)

Machine learning approaches often require training and evaluation datasets with a clear separation between positive and negative examples. This risks simplifying and even obscuring the inherent subjectivity present in many tasks. Preserving such variance in content and diversity in datasets is often expensive and laborious. This is especially troubling when building safety datasets for conversational AI systems, as safety is both socially and culturally situated. To demonstrate this crucial aspect of conversational AI safety, and to facilitate in-depth model performance analyses, we introduce the DICES (Diversity In Conversational AI Evaluation for Safety) dataset that contains fine-grained demographic information about raters, high replication of ratings per item to ensure statistical power for analyses, and encodes rater votes as distributions across different demographics to allow for in-depth explorations of different aggregation strategies. In short, the DICES dataset enables the observation and measurement of variance, ambiguity, and diversity in the context of conversational AI safety. We also illustrate how the dataset offers a basis for establishing metrics to show how raters' ratings can intersects with demographic categories such as racial/ethnic groups, age groups, and genders. The goal of DICES is to be used as a shared resource and benchmark that respects diverse perspectives during safety evaluation of conversational AI systems.