# DisaBench: A Participatory Evaluation Framework for Disability Harms in Language Models

arXiv: https://arxiv.org/abs/2605.12702 · HTML: https://mlsanity.com/paper/2605.12702/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 11 of 11)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- ["Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled People](https://mlsanity.com/paper/2410.03448/) (2024)

## Abstract (arXiv)

General-purpose safety benchmarks for large language models do not adequately evaluate disability-related harms. We introduce DisaBench: a taxonomy of twelve disability harm categories co-created with people with disabilities and red teaming experts, a taxonomy-driven evaluation methodology that pairs benign and adversarial prompts across seven life domains, and a dataset of 175 prompts with human-annotated labels on 525 prompt-response pairs. Annotation by four evaluators with lived disability experience reveals three findings: harm rates vary sharply by disability type and will compound in non-text modalities, terminology-driven harm is culturally and temporally bound rather than universally assessable, and standard safety evaluation catches overt failures while missing the subtle harms that only domain expertise can recognize. Disability harm is simultaneously personal, intersectional, and community-defined: it cannot be isolated from the full context of who a person is, and general-purpose benchmarks systematically miss it. We will release the dataset, taxonomy, and methodology via Hugging Face and an open-source red teaming framework for direct integration into existing safety pipelines with no additional infrastructure.