# Characterizing Selective Refusal Bias in Large Language Models

arXiv: https://arxiv.org/abs/2510.27087 · HTML: https://mlsanity.com/paper/2510.27087/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal](https://mlsanity.com/paper/2406.14598/) (2024)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [The Art of Saying No: Contextual Noncompliance in Language Models](https://mlsanity.com/paper/2407.12043/) (2024)
- [Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://mlsanity.com/paper/2005.02439/) (2020)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion](https://mlsanity.com/paper/2403.07865/) (2024)
- [Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis](https://mlsanity.com/paper/2406.10794/) (2024)
- [Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles](https://mlsanity.com/paper/2408.04686/) (2024)
- [From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasets](https://mlsanity.com/paper/2404.17874/) (2024)
- [Intersectional Bias in Hate Speech and Abusive Language Datasets](https://mlsanity.com/paper/2005.05921/) (2020)
- [textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities](https://mlsanity.com/paper/2410.06967/) (2024)
- [Evaluating Biased Attitude Associations of Language Models in an Intersectional Context](https://mlsanity.com/paper/2307.03360/) (2023)
- [Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs](https://mlsanity.com/paper/2502.01926/) (2025)
- [LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users](https://mlsanity.com/paper/2406.17737/) (2024)
- [Analyzing Hate Speech Data along Racial, Gender and Intersectional Axes](https://mlsanity.com/paper/2205.06621/) (2022)
- [Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment](https://mlsanity.com/paper/2406.11285/) (2024)

## Built on by (most cited first, 4 of 4)

- [Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making](https://mlsanity.com/paper/2605.24319/) (2026)
- [State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Models](https://mlsanity.com/paper/2512.13762/) (2025)
- [Position: Fairness Failure in Generative Models is an Evaluation Problem](https://mlsanity.com/paper/2608.16974/) (2026)
- [When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation](https://mlsanity.com/paper/2609.22102/) (2026)

## Abstract (arXiv)

Safety guardrails in large language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.