# Do Large Language Models Reflect Demographic Pluralism in Safety?

arXiv: https://arxiv.org/abs/2602.07376 · HTML: https://mlsanity.com/paper/2602.07376/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://mlsanity.com/paper/2306.11247/) (2023)
- [The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models](https://mlsanity.com/paper/2405.04623/) (2024)
- [Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model](https://mlsanity.com/paper/2501.02790/) (2025)
- [Too Helpful, Too Harmless, Too Honest or Just Right?](https://mlsanity.com/paper/2509.08486/) (2025)
- [Should LLM Safety Be More Than Refusing Harmful Instructions?](https://mlsanity.com/paper/2506.02442/) (2025)
- [Steering Towards Fairness: Mitigating Political Bias in LLMs](https://mlsanity.com/paper/2508.08846/) (2025)
- [Truth, Trust, and Trouble: Medical AI on the Edge](https://mlsanity.com/paper/2507.02983/) (2025)

## Built on by (most cited first, 1 of 1)

- [AlignCultura: Towards Culturally Aligned Large Language Models?](https://mlsanity.com/paper/2604.19016/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) safety is inherently pluralistic, reflecting variations in moral norms, cultural expectations, and demographic contexts. Yet, existing alignment datasets such as ANTHROPIC-HH and DICES rely on demographically narrow annotator pools, overlooking variation in safety perception across communities. Demo-SafetyBench addresses this gap by modeling demographic pluralism directly at the prompt level, decoupling value framing from responses. In Stage I, prompts from DICES are reclassified into 14 safety domains (adapted from BEAVERTAILS) using Mistral 7B-Instruct-v0.3, retaining demographic metadata and expanding low-resource domains via Llama-3.1-8B-Instruct with SimHash-based deduplication, yielding 43,050 samples. In Stage II, pluralistic sensitivity is evaluated using LLMs-as-Raters-Gemma-7B, GPT-4o, and LLaMA-2-7B-under zero-shot inference. Balanced thresholds (delta = 0.5, tau = 10) achieve high reliability (ICC = 0.87) and low demographic sensitivity (DS = 0.12), confirming that pluralistic safety evaluation can be both scalable and demographically robust.