# HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models

arXiv: https://arxiv.org/abs/2211.02882 · HTML: https://mlsanity.com/paper/2211.02882/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 5 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [A Simple Method for Commonsense Reasoning](https://mlsanity.com/paper/1806.02847/) (2018)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [Mitigating Political Bias in Language Models Through Reinforced Calibration](https://mlsanity.com/paper/2104.14795/) (2021)
- [Demoting Racial Bias in Hate Speech Detection](https://mlsanity.com/paper/2005.12246/) (2020)
- [Unmasking the Mask -- Evaluating Social Biases in Masked Language Models](https://mlsanity.com/paper/2104.07496/) (2021)
- [Investigating African-American Vernacular English in Transformer-Based Text Generation](https://mlsanity.com/paper/2010.02510/) (2020)
- [Examining Gender Bias in Languages with Grammatical Gender](https://mlsanity.com/paper/1909.02224/) (2019)
- [Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models](https://mlsanity.com/paper/2112.07447/) (2021)
- [Gender Bias Hidden Behind Chinese Word Embeddings: The Case of Chinese Adjectives](https://mlsanity.com/paper/2106.00181/) (2021)

## Built on by (most cited first, 8 of 8)

- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models](https://mlsanity.com/paper/2406.10130/) (2024)
- [Uncovering and Categorizing Social Biases in Text-to-SQL](https://mlsanity.com/paper/2305.16253/) (2023)
- [Detecting Spoilers in Movie Reviews with External Movie Knowledge and User Networks](https://mlsanity.com/paper/2304.11411/) (2023)
- [Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge](https://mlsanity.com/paper/2506.12574/) (2025)

## Abstract (arXiv)

Fairness has become a trending topic in natural language processing (NLP), which addresses biases targeting certain social groups such as genders and religions. However, regional bias in language models (LMs), a long-standing global discrimination problem, still remains unexplored. This paper bridges the gap by analysing the regional bias learned by the pre-trained language models that are broadly used in NLP tasks. In addition to verifying the existence of regional bias in LMs, we find that the biases on regional groups can be strongly influenced by the geographical clustering of the groups. We accordingly propose a HiErarchical Regional Bias evaluation method (HERB) utilising the information from the sub-region clusters to quantify the bias in pre-trained LMs. Experiments show that our hierarchical metric can effectively evaluate the regional bias with respect to comprehensive topics and measure the potential regional bias that can be propagated to downstream tasks. Our codes are available at https://github.com/Bernard-Yang/HERB.