mlsanityData to 7 Oct 2026

BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization

arXiv 2407.13928 ↗First posted 18 Jul 2024cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large Language Models (LLMs) have become pivotal in advancing natural language processing, yet their potential to perpetuate biases poses significant concerns. This paper introduces a new framework employing Direct Preference Optimization (DPO) to mitigate gender, racial, and religious biases in LLM-generated English text. By developing a loss function that favors less biased over biased completions, our approach cultivates a preference for respectful and non-discriminatory language in LLMs. We also contribute a manually designed dataset for training LLMs to recognize and correct biases. This dataset encompasses a diverse range of prompts paired with both biased and unbiased completions. Implementing this approach on the Microsoft Phi-2 model, we demonstrate substantial reductions in biased outputs as our model outperforms the baseline model on almost all bias benchmarks. Our model also achieves better performance compared to other open-source models on most benchmarks. By reducing biases in the language generated by the model, our study marks a significant step towards developing more ethical and socially responsible LLMs. We publicly release BiasDPO dataset on HuggingFace.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 19 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

10 of 19 references.

References · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  8. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  9. 2024Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypescited by 58
  10. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 70
  2. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  3. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 37
  4. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  5. 2025Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMscited by 10
  6. 2026Position: LLMs Must Use Functor-Based and RAG-Driven Bias Mitigation for Fairnesscited by 8
  7. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  8. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  9. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  10. 2025Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMscited by 3
  11. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  12. 2026Self-Debias: Self-correcting for Debiasing Large Language Modelscited by 2
6 more
  1. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Modelscited by 1
  2. 2025Can SAEs reveal and mitigate racial biases of LLMs in healthcare?cited by 1
  3. 2025Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Modelscited by 0
  4. 2026Faithful-Patchscopes: Understanding and Mitigating Model Bias in Hidden Representations Explanation of Large Language Modelscited by 0
  5. 2026Mitigating LLM biases toward spurious social contexts using direct preference optimizationcited by 0
  6. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds
  2. 2024A Multi-LLM Debiasing Framework
  3. 2024Evaluating Nuanced Bias in Large Language Model Free Response Answers
  4. 2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs
  5. 2025Mitigating Social Bias in English and Urdu Language Models Using PRM-Guided Candidate Selection and Sequential Refinement
  6. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  7. 2025Silenced Biases: The Dark Side LLMs Learned to Refuse
  8. 2024Disclosure and Mitigation of Gender Bias in LLMs