mlsanityData to 7 Oct 2026

What's Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMs

arXiv 2502.19749 ↗First posted 27 Feb 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often exhibit social biases inherited from their training data. While existing benchmarks evaluate bias by term-based mode through direct term associations between demographic terms and bias terms, LLMs have become increasingly adept at avoiding biased responses, leading to seemingly low levels of bias. However, biases persist in subtler, contextually hidden forms that traditional benchmarks fail to capture. We introduce the Description-based Bias Benchmark (DBB), a novel dataset designed to assess bias at the semantic level that bias concepts are hidden within naturalistic, subtly framed contexts in real-world scenarios rather than superficial terms. We analyze six state-of-the-art LLMs, revealing that while models reduce bias in response at the term level, they continue to reinforce biases in nuanced settings. Data, code, and results are available at https://github.com/JP-25/Description-based-Bias-Benchmark.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

23 of 37 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
11 more
  1. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  2. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  3. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  4. 2024Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypescited by 58
  5. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 55
  6. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 39
  7. 2024Gender Bias in Large Language Models across Multiple Languagescited by 35
  8. 2020Unsupervised Discovery of Implicit Gender Biascited by 34
  9. 2023Social Bias Probing: Fairness Benchmarking for Language Modelscited by 29
  10. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  11. 2023Measuring Gender Bias in West Slavic Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzlescited by 1
  2. 2026Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisionscited by 1
  3. 2026The Straight and Narrow: Do LLMs Possess an Internal Moral Path?cited by 0
  4. 2026Knowing Bias, Doing Better: Mitigating Social Bias in LLMs via Know-Bias Neuron Enhancementcited by 0
  5. 2026Debias It Yourself: Teaching LLMs Cognitive Bias Mitigation Interventionscited by 0
  6. 2026Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Informationcited by 0