mlsanityData to 7 Oct 2026

Mitigating Language-Dependent Ethnic Bias in BERT

arXiv 2109.05704 ↗First posted 13 Sep 2021cs.CL cs.AIcited by 87

Not on the map. Why?

Abstract (arXiv)

BERT and other large-scale language models (LMs) contain gender and racial bias. They also exhibit other dimensions of social bias, most of which have not been studied in depth, and some of which vary depending on the language. In this paper, we study ethnic bias and how it varies across languages by analyzing and mitigating ethnic bias in monolingual BERT for English, German, Spanish, Korean, Turkish, and Chinese. To observe and quantify ethnic bias, we develop a novel metric called Categorical Bias score. Then we propose two methods for mitigation; first using a multilingual model, and second using contextual word alignment of two monolingual models. We compare our proposed methods with monolingual BERT and show that these methods effectively alleviate the ethnic bias. Which of the two methods works better depends on the amount of NLP resources available for that language. We additionally experiment with Arabic and Greek to verify that our proposed methods work for a wider variety of languages.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

30 of 55 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  8. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  9. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  10. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  11. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  12. 2019Measuring Bias in Contextualized Word Representationscited by 275
18 more
  1. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  2. 2018Gender Bias in Neural Natural Language Processingcited by 210
  3. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  4. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  5. 2020Towards Debiasing Sentence Representationscited by 170
  6. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  7. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  8. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  9. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  10. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 94
  11. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  12. 2020Gender Bias in Multilingual Embeddings and Cross-Lingual Transfercited by 55
  13. 2019Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsingcited by 46
  14. 2020Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learningcited by 44
  15. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  16. 2020Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigationcited by 36
  17. 2019Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectorscited by 31
  18. 2020Mitigating Gender Bias Amplification in Distribution by Posterior Regularizationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 25; 2024: 25; 2025: 21; 2026: 4.

87 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  3. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  4. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  5. 2024Systematic Biases in LLM Simulations of Debatescited by 98
  6. 2023A Survey on Fairness in Large Language Modelscited by 91
  7. 2024Large Language Models are Geographically Biasedcited by 91
  8. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  9. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  10. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  11. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  12. 2023Comparing Biases and the Impact of Multilingual Training across Multiple Languagescited by 36
38 more
  1. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  2. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  3. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  4. 2023Are Emily and Greg Still More Employable than Lakisha and Jamal? Investigating Algorithmic Hiring Bias in the Era of ChatGPTcited by 23
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  6. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  7. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  8. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
  9. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  10. 2024Large Language Models Still Exhibit Bias in Long Textcited by 16
  11. 2025Robustly Improving LLM Fairness in Realistic Settings via Interpretabilitycited by 16
  12. 2023Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniquescited by 14
  13. 2023DisCGen: A Framework for Discourse-Informed Counterspeech Generationcited by 14
  14. 2023CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Biascited by 13
  15. 2024Do Multilingual Large Language Models Mitigate Stereotype Bias?cited by 13
  16. 2023D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Biascited by 12
  17. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  18. 2024Fairness Definitions in Language Models Explainedcited by 10
  19. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  20. 2025A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Biascited by 10
  21. 2023SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddingscited by 9
  22. 2023This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Modelscited by 9
  23. 2024No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Userscited by 9
  24. 2022Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Biascited by 8
  25. 2023In-Depth Look at Word Filling Societal Bias Measurescited by 8
  26. 2025Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategiescited by 8
  27. 2022Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modelingcited by 6
  28. 2022An Analysis of Social Biases Present in BERT Variants Across Multiple Languagescited by 6
  29. 2022Power of Explanations: Towards automatic debiasing in hate speech detectioncited by 5
  30. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  31. 2024FairMonitor: A Dual-framework for Detecting Stereotypes and Biases in Large Language Modelscited by 5
  32. 2024GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanationscited by 5
  33. 2024An Active Learning Framework for Inclusive Generation by Large Language Modelscited by 5
  34. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  35. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4
  36. 2024Exploring Bengali Religious Dialect Biases in Large Language Models with Evaluation Perspectivescited by 4
  37. 2024BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Modelscited by 4
  38. 2022Visualizing Deep Neural Networks with Topographic Activation Mapscited by 3

50 most cited of 87 All 87 (JSON)