# Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions

arXiv: https://arxiv.org/abs/2410.02584 · HTML: https://mlsanity.com/paper/2410.02584/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agents](https://mlsanity.com/paper/2403.08715/) (2024)
- [On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization](https://mlsanity.com/paper/2405.16455/) (2024)
- [Small Language Model Can Self-correct](https://mlsanity.com/paper/2401.07301/) (2024)
- [Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs](https://mlsanity.com/paper/2404.10160/) (2024)
- [Prompting Fairness: Integrating Causality to Debias Large Language Models](https://mlsanity.com/paper/2403.08743/) (2024)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2303.02891/) (2023)

## Built on by (most cited first, 30 of 51)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Unmasking Conversational Bias in AI Multiagent Systems](https://mlsanity.com/paper/2501.14844/) (2025)
- [Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancy](https://mlsanity.com/paper/2503.09639/) (2025)
- [Mind the (Belief) Gap: Group Identity in the World of LLMs](https://mlsanity.com/paper/2503.02016/) (2025)
- [Persuasion at Play: Understanding Misinformation Dynamics in Demographic-Aware Human-LLM Interactions](https://mlsanity.com/paper/2503.02038/) (2025)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Implicit Bias in LLMs: A Survey](https://mlsanity.com/paper/2503.02776/) (2025)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions](https://mlsanity.com/paper/2511.11789/) (2025)
- [Position: LLMs Can be Good Tutors in English Education](https://mlsanity.com/paper/2502.05467/) (2025)
- [The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities](https://mlsanity.com/paper/2508.05525/) (2025)
- [Emergent Bias and Fairness in Multi-Agent Decision Systems](https://mlsanity.com/paper/2512.16433/) (2025)
- [Replicating Human Motivated Reasoning Studies with LLMs](https://mlsanity.com/paper/2601.16130/) (2026)
- [Veracity Bias and Beyond: Uncovering LLMs' Hidden Beliefs in Problem-Solving Reasoning](https://mlsanity.com/paper/2505.16128/) (2025)
- [MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered](https://mlsanity.com/paper/2507.01019/) (2025)
- [Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theory](https://mlsanity.com/paper/2510.13902/) (2025)
- [Emergence of Biased Consensus in Multi-Agent LLM Debates](https://mlsanity.com/paper/2608.02827/) (2026)
- [Interacting Large Language Model Agents. Interpretable Models and Social Learning](https://mlsanity.com/paper/2411.01271/) (2024)
- [Are Human Interactions Replicable by Generative Agents? A Case Study on Pronoun Usage in Hierarchical Interactions](https://mlsanity.com/paper/2501.15283/) (2025)
- [Breaking Down Bias: On The Limits of Generalizable Pruning Strategies](https://mlsanity.com/paper/2502.07771/) (2025)
- [Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengths](https://mlsanity.com/paper/2506.02481/) (2025)
- [Evaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards](https://mlsanity.com/paper/2506.04975/) (2025)
- [BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responses](https://mlsanity.com/paper/2510.00232/) (2025)
- [The Social Cost of Intelligence: Emergence, Propagation, and Amplification of Stereotypical Bias in Multi-Agent Systems](https://mlsanity.com/paper/2510.10943/) (2025)
- [LLM-enabled Applications Require System-Level Threat Monitoring](https://mlsanity.com/paper/2602.19844/) (2026)
- [Social Dynamics as Critical Vulnerabilities that Undermine Objective Decision-Making in LLM Collectives](https://mlsanity.com/paper/2604.06091/) (2026)
- [Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models](https://mlsanity.com/paper/2506.08593/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs are susceptible to societal biases due to their exposure to human-generated data. Given that LLMs are being used to gain insights into various societal aspects, it is essential to mitigate these biases. To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases. We begin by creating a dataset of scenarios where implicit gender biases might arise, and subsequently develop a metric to assess the presence of biases. Our empirical analysis reveals that LLMs generate outputs characterized by strong implicit bias associations (>= 50% of the time). Furthermore, these biases tend to escalate following multi-agent interactions. To mitigate them, we propose two strategies: self-reflection with in-context examples (ICE); and supervised fine-tuning. Our research demonstrates that both methods effectively mitigate implicit biases, with the ensemble of fine-tuning and self-reflection proving to be the most successful.