mlsanityData to 7 Oct 2026

Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach

arXiv 2412.11679 ↗First posted 16 Dec 2024cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The use of language models (LMs) has increased considerably in recent years, and the biases and stereotypes in training data that are reflected in the LM outputs are causing social problems. In this paper, inspired by the task arithmetic, we propose the ``Bias Vector'' method for the mitigation of these LM biases. The Bias Vector method does not require manually created debiasing data. The three main steps of our approach involve: (1) continual training the pre-trained LMs on biased data using masked language modeling; (2) constructing the Bias Vector as the difference between the weights of the biased LMs and those of pre-trained LMs; and (3) subtracting the Bias Vector from the weights of the pre-trained LMs for debiasing. We evaluated the Bias Vector method on the SEAT across three LMs and confirmed an average improvement of 0.177 points. We demonstrated that the Bias Vector method does not degrade the LM performance on downstream tasks in the GLUE benchmark. In addition, we examined the impact of scaling factors, which control the magnitudes of Bias Vectors, with effect sizes on the SEAT and conducted a comprehensive evaluation of our debiased LMs across both the SEAT and GLUE benchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

33 of 38 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 613
21 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  3. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  4. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  7. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  8. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  9. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  10. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  11. 2020Towards Debiasing Sentence Representationscited by 170
  12. 2018Learning Gender-Neutral Word Embeddingscited by 169
  13. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  14. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  15. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  16. 2019Does Gender Matter? Towards Fairness in Dialogue Systemscited by 89
  17. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  18. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  19. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  20. 2020Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigationcited by 36
  21. 2023Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Taskcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 3.

12 papers.

  1. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  2. 2026Bias Beyond Borders: Political Ideology Evaluation and Steering in Multilingual LLMscited by 4
  3. 2025Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Datasetcited by 2
  4. 2025Fairness Evaluation and Inference Level Mitigation in LLMscited by 2
  5. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  6. 2025Exploring and Leveraging Class Vectors for Classifier Editingcited by 1
  7. 2026A Heuristic Perspective on Debiasing Language Modelscited by 1
  8. 2025Transferring Visual Explainability of Self-Explaining Models to Prediction-Only Models without Additional Trainingcited by 0
  9. 2025Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFTcited by 0
  10. 2025Uncovering Implicit Bias in Large Language Models with Concept Learning Datasetcited by 0
  11. 2025An Empirical Survey of Model Merging Algorithms for Social Bias Mitigationcited by 0
  12. 2026MergeRec: Model Merging for Data-Isolated Cross-Domain Sequential Recommendationcited by 0