mlsanityData to 7 Oct 2026

Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions

arXiv 2410.02584 ↗First posted 3 Oct 2024cs.CL cs.CYcited by 50

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs are susceptible to societal biases due to their exposure to human-generated data. Given that LLMs are being used to gain insights into various societal aspects, it is essential to mitigate these biases. To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases. We begin by creating a dataset of scenarios where implicit gender biases might arise, and subsequently develop a metric to assess the presence of biases. Our empirical analysis reveals that LLMs generate outputs characterized by strong implicit bias associations (>= 50% of the time). Furthermore, these biases tend to escalate following multi-agent interactions. To mitigate them, we propose two strategies: self-reflection with in-context examples (ICE); and supervised fine-tuning. Our research demonstrates that both methods effectively mitigate implicit biases, with the ensemble of fine-tuning and self-reflection proving to be the most successful.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 26 of its 57 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

26 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  8. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  9. 2018Gender Bias in Coreference Resolutioncited by 438
  10. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  12. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
13 more
  1. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  2. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  3. 2018Gender Bias in Neural Natural Language Processingcited by 210
  4. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  5. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  6. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  8. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  9. 2024Small Language Model Can Self-correctcited by 23
  10. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  11. 2024Prompting Fairness: Integrating Causality to Debias Large Language Modelscited by 20
  12. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 19
  13. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedbackcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 31; 2026: 18.

50 papers.

Sort:

By area:

  1. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  4. 2026Large Language Model Reasoning Failurescited by 20
  5. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 16
  6. 2025Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancycited by 15
  7. 2025Mind the (Belief) Gap: Group Identity in the World of LLMscited by 12
  8. 2025Persuasion at Play: Understanding Misinformation Dynamics in Demographic-Aware Human-LLM Interactionscited by 10
  9. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  10. 2025Implicit Bias in LLMs: A Surveycited by 8
  11. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  12. 2025From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactionscited by 7
38 more
  1. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6
  2. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  3. 2025Emergent Bias and Fairness in Multi-Agent Decision Systemscited by 5
  4. 2026Replicating Human Motivated Reasoning Studies with LLMscited by 5
  5. 2025Veracity Bias and Beyond: Uncovering LLMs' Hidden Beliefs in Problem-Solving Reasoningcited by 3
  6. 2025MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncoveredcited by 3
  7. 2025Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theorycited by 3
  8. 2024Interacting Large Language Model Agents. Interpretable Models and Social Learningcited by 2
  9. 2025Are Human Interactions Replicable by Generative Agents? A Case Study on Pronoun Usage in Hierarchical Interactionscited by 2
  10. 2025Breaking Down Bias: On The Limits of Generalizable Pruning Strategiescited by 2
  11. 2025Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengthscited by 2
  12. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  13. 2025The Social Cost of Intelligence: Emergence, Propagation, and Amplification of Stereotypical Bias in Multi-Agent Systemscited by 2
  14. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  15. 2026Social Dynamics as Critical Vulnerabilities that Undermine Objective Decision-Making in LLM Collectivescited by 2
  16. 2026Emergence of Biased Consensus in Multi-Agent LLM Debatescited by 2
  17. 2025Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Modelscited by 1
  18. 2025EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Normscited by 1
  19. 2025I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluationscited by 1
  20. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  21. 2025Bias Mitigation Agent: Optimizing Source Selection for Fair and Balanced Knowledge Retrievalcited by 1
  22. 2025"The Dentist is an involved parent, the bartender is not": Revealing Implicit Biases in QA with Implicit BBQcited by 1
  23. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  24. 2026Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibilitycited by 1
  25. 2026Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent Systemscited by 1
  26. 2025Integrating gender inclusivity into large language models via instruction tuningcited by 0
  27. 2025Two-Stage Voting for Robust and Efficient Suicide Risk Detection on Social Mediacited by 0
  28. 2025The Curious Case of Curiosity across Human Cultures and LLMscited by 0
  29. 2025Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relationscited by 0
  30. 2026Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domainscited by 0
  31. 2026Exploring Implicit Perspectives on Autism in Large Language Models Through Multi-Agent Simulationscited by 0
  32. 2026TransportAgents: a multi-agents LLM framework for traffic accident severity predictioncited by 0
  33. 2026ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cuescited by 0
  34. 2026Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Reviewcited by 0
  35. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Modelscited by 0
  36. 2026Beyond Outcome Gaps: Process-Aware Fairness Diagnosis for LLM-based Multi-Agent Decision Systemscited by 0
  37. 2026Value-Preserving Architectures for Agentic AI Systemscited by 0
  38. 2026Bias Amplification in Multi-Agent Network: How Biased Agents Shape Opinions and Rhetoriccited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models
  2. 2024Systematic Biases in LLM Simulations of Debates
  3. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection
  4. 2024Disclosure and Mitigation of Gender Bias in LLMs
  5. 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models
  6. 2024Investigating Bias Representations in Llama 2 Chat via Activation Steering
  7. 2024A Multi-LLM Debiasing Framework
  8. 2024White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMs