mlsanityData to 14 Oct 2026

From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddings

arXiv 2402.11512 ↗First posted 18 Feb 2024cs.CL cs.CYcited by 4

Not on the map. Why?

Abstract (arXiv)

Embeddings play a pivotal role in the efficacy of Large Language Models. They are the bedrock on which these models grasp contextual relationships and foster a more nuanced understanding of language and consequently perform remarkably on a plethora of complex tasks that require a fundamental understanding of human language. Given that these embeddings themselves often reflect or exhibit bias, it stands to reason that these models may also inadvertently learn this bias. In this work, we build on the seminal previous work and propose DeepSoftDebias, an algorithm that uses a neural network to perform 'soft debiasing'. We exhaustively evaluate this algorithm across a variety of SOTA datasets, accuracy metrics, and challenging NLP tasks. We find that DeepSoftDebias outperforms the current state-of-the-art methods at reducing bias across gender, race, and religion.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

31 of 43 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
19 more
  1. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  2. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  3. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  5. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  6. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  7. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  8. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  9. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  10. 2020Towards Debiasing Sentence Representationscited by 171
  11. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 147
  12. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  13. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  14. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  15. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  16. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  17. 2018Native Language Cognate Effects on Second Language Lexical Choicecited by 11
  18. 2023FineDeb: A Debiasing Framework for Language Modelscited by 7
  19. 2019Sentiment Analysis on IMDB Movie Comments and Twitter Data by Machine Learning and Vector Space Techniquescited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Large Language Model Safety: A Holistic Surveycited by 59
  2. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
  3. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  4. 2025KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Modelscited by 3