mlsanityData to 14 Oct 2026

Characterizing Selective Refusal Bias in Large Language Models

arXiv 2510.27087 ↗First posted 31 Oct 2025cs.CL cs.CYcited by 4

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 22 of its 45 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

22 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
  2. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion12 Mar

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Ethical and social risks of harm from Language Modelscited by 966
  3. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  4. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  5. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  6. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  7. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  8. 2020Contextualizing Hate Speech Classifiers with Post-hoc Explanationcited by 84
  9. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  10. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  11. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 64
  12. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
8 more
  1. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 25
  2. 2020Intersectional Bias in Hate Speech and Abusive Language Datasetscited by 22
  3. 2024textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalitiescited by 22
  4. 2023Evaluating Biased Attitude Associations of Language Models in an Intersectional Contextcited by 20
  5. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  6. 2024LLM Targeted Underperformance Disproportionately Impacts Vulnerable Userscited by 15
  7. 2022Analyzing Hate Speech Data along Racial, Gender and Intersectional Axescited by 7
  8. 2024Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-makingcited by 2
  2. 2025State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Modelscited by 0
  3. 2026Position: Fairness Failure in Generative Models is an Evaluation Problemcited by 0
  4. 2026When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  2. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models
  3. 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
  4. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
  5. 2025Are LLMs Good Safety Agents or a Propaganda Engine?
  6. 2025Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences
  7. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles
  8. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models