# Perturbation Sensitivity Analysis to Detect Unintended Model Biases

arXiv: https://arxiv.org/abs/1910.04210 · HTML: https://mlsanity.com/paper/1910.04210/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 30 of 74)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [Perturbation Augmentation for Fairer NLP](https://mlsanity.com/paper/2205.12586/) (2022)
- [Nationality Bias in Text Generation](https://mlsanity.com/paper/2302.02463/) (2023)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022)
- [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022)
- [Comparing Biases and the Impact of Multilingual Training across Multiple Languages](https://mlsanity.com/paper/2305.11242/) (2023)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP](https://mlsanity.com/paper/2106.06830/) (2021)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation](https://mlsanity.com/paper/2106.05532/) (2021)
- [Brittlebench: Quantifying LLM robustness via prompt sensitivity](https://mlsanity.com/paper/2603.13285/) (2026)
- [Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models](https://mlsanity.com/paper/2206.03945/) (2022)
- [A Human Rights-Based Approach to Responsible AI](https://mlsanity.com/paper/2210.02667/) (2022)
- [Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval](https://mlsanity.com/paper/2212.10526/) (2022)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- [Fairness-aware Class Imbalanced Learning](https://mlsanity.com/paper/2109.10444/) (2021)
- [Interpreting the Robustness of Neural NLP Models to Textual Perturbations](https://mlsanity.com/paper/2110.07159/) (2021)
- [Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles](https://mlsanity.com/paper/2308.04346/) (2023)
- [The "Colonial Impulse" of Natural Language Processing: An Audit of Bengali Sentiment Analysis Tools and Their Identity-based Biases](https://mlsanity.com/paper/2401.10535/) (2024)
- [Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models](https://mlsanity.com/paper/2104.07505/) (2021)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)

## Abstract (arXiv)

Data-driven statistical Natural Language Processing (NLP) techniques leverage large amounts of language data to build models that can understand language. However, most language data reflect the public discourse at the time the data was produced, and hence NLP models are susceptible to learning incidental associations around named referents at a particular point in time, in addition to general linguistic meaning. An NLP system designed to model notions such as sentiment and toxicity should ideally produce scores that are independent of the identity of such entities mentioned in text and their social associations. For example, in a general purpose sentiment analysis system, a phrase such as I hate Katy Perry should be interpreted as having the same sentiment as I hate Taylor Swift. Based on this idea, we propose a generic evaluation framework, Perturbation Sensitivity Analysis, which detects unintended model biases related to named entities, and requires no new annotations or corpora. We demonstrate the utility of this analysis by employing it on two different NLP models --- a sentiment model and a toxicity model --- applied on online comments in English language from four different genres.