mlsanityData to 7 Oct 2026

Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs

arXiv 2404.10160 ↗First posted 15 Apr 2024cs.AIcited by 22

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs on BBQ and our datasets demonstrate the effectiveness of our approach in bias mitigation. Our source code and datasets are available at texttthttps://anonymous.4open.science/r/RLDF-E344.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 54 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

44 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection21 Mar
  2. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models19 Feb
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
27 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  5. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 281
  6. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 246
  7. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  8. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  9. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  10. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  11. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  12. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 119
  13. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  14. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  15. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  16. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  17. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  18. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48
  19. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  20. 2022Fairness Reprogrammingcited by 33
  21. 2023Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Modelscited by 32
  22. 2023Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architecturescited by 25
  23. 2024Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLMcited by 21
  24. 2024Multi-role Consensus through LLMs Discussions for Vulnerability Detectioncited by 19
  25. 2024Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answeringcited by 18
  26. 2024Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like Onecited by 9
  27. 2017Learning to Learn from Weak Supervision by Full Supervisioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 50
  2. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  3. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Informationcited by 25
  4. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  5. 2024PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximizationcited by 17
  6. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  7. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  8. 2024Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Frameworkcited by 8
  9. 2025EcoAlign: An Economically Rational Framework for Efficient LVLM Alignmentcited by 5
  10. 2025A Benchmark for Scalable Oversight Protocolscited by 2
  11. 2025MPF: Aligning and Debiasing Language Models post Deployment via Multi Perspective Fusioncited by 2
  12. 2025Contemplative Artificial Intelligencecited by 1
10 more
  1. 2025A Reward-driven Automated Webshell Malicious-code Generator for Red-teamingcited by 1
  2. 2025Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learningcited by 1
  3. 2026Identity-Robust Language Model Generation via Content Integrity Preservationcited by 1
  4. 2025DMRL: Data- and Model-aware Reward Learning for Data Extractioncited by 0
  5. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Searchcited by 0
  6. 2025Information-Dense Reasoning for Efficient and Auditable Security Alert Triagecited by 0
  7. 2026ShellForge: Adversarial Co-Evolution of Webshell Generation and Multi-View Detection for Robust Webshell Defensecited by 0
  8. 2026Membership Inference for Contrastive Pre-training Models with Text-only PII Queriescited by 0
  9. 2026Where are the Humans? A Scoping Review of Fairness in Multi-agent AI Systemscited by 0
  10. 2026Same Chart, Different Story: Bias in Vision-Language Chart Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Detecting Prefix Bias in LLM-based Reward Models
  2. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback
  3. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models
  4. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs
  7. 2026Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoning
  8. 2024Systematic Biases in LLM Simulations of Debates