# Assessing Demographic Bias in Named Entity Recognition

arXiv: https://arxiv.org/abs/2008.03415 · HTML: https://mlsanity.com/paper/2008.03415/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal](https://mlsanity.com/paper/2203.12574/) (2022)
- [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://mlsanity.com/paper/2406.11565/) (2024)
- [Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency](https://mlsanity.com/paper/2105.08667/) (2021)
- [Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictions](https://mlsanity.com/paper/2407.05271/) (2024)
- [In the Name of Fairness: Assessing the Bias in Clinical Record De-identification](https://mlsanity.com/paper/2305.11348/) (2023)
- [Behind the Mask: Demographic bias in name detection for PII masking](https://mlsanity.com/paper/2205.04505/) (2022)
- [Examining risks of racial biases in NLP tools for child protective services](https://mlsanity.com/paper/2305.19409/) (2023)
- [The Lifecycle of "Facts": A Survey of Social Bias in Knowledge Graphs](https://mlsanity.com/paper/2210.03353/) (2022)
- [TweetNERD -- End to End Entity Linking Benchmark for Tweets](https://mlsanity.com/paper/2210.08129/) (2022)
- [A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models](https://mlsanity.com/paper/2212.12799/) (2022)
- [Crowdsourcing on Sensitive Data with Privacy-Preserving Text Rewriting](https://mlsanity.com/paper/2303.03053/) (2023)
- [Alt-Text with Context: Improving Accessibility for Images on Twitter](https://mlsanity.com/paper/2305.14779/) (2023)
- [Automated Testing and Improvement of Named Entity Recognition Systems](https://mlsanity.com/paper/2308.07937/) (2023)
- [BEADs: Bias Evaluation Across Domains](https://mlsanity.com/paper/2406.04220/) (2024)
- [NERDA-Con: Extending NER models for Continual Learning -- Integrating Distinct Tasks and Updating Distribution Shifts](https://mlsanity.com/paper/2206.14607/) (2022)
- [CTM -- A Model for Large-Scale Multi-View Tweet Topic Classification](https://mlsanity.com/paper/2205.01603/) (2022)
- [Robust Candidate Generation for Entity Linking on Short Social Media Texts](https://mlsanity.com/paper/2210.07472/) (2022)
- [Towards Fair and Explainable AI using a Human-Centered AI Approach](https://mlsanity.com/paper/2306.07427/) (2023)
- [Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation](https://mlsanity.com/paper/2508.16762/) (2025)
- [Large Language Models for Geolocation Extraction in Humanitarian Crisis Response](https://mlsanity.com/paper/2602.08872/) (2026)
- [#maskUp: Selective Attribute Encryption for Sensitive Vocalization for English language on Social Media Platforms](https://mlsanity.com/paper/2211.08653/) (2022)
- [Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation](https://mlsanity.com/paper/2404.01334/) (2024)

## Abstract (arXiv)

Named Entity Recognition (NER) is often the first step towards automated Knowledge Base (KB) generation from raw text. In this work, we assess the bias in various Named Entity Recognition (NER) systems for English across different demographic groups with synthetically generated corpora. Our analysis reveals that models perform better at identifying names from specific demographic groups across two datasets. We also identify that debiased embeddings do not help in resolving this issue. Finally, we observe that character-based contextualized word representation models such as ELMo results in the least bias across demographics. Our work can shed light on potential biases in automated KB generation due to systematic exclusion of named entities belonging to certain demographics.