# MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered

arXiv: https://arxiv.org/abs/2507.01019 · HTML: https://mlsanity.com/paper/2507.01019/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 13 of 13)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [Large Language Models Still Exhibit Bias in Long Text](https://mlsanity.com/paper/2410.17519/) (2024)

## Built on by (most cited first, 3 of 3)

- [Free-MAD: Consensus-Free Multi-Agent Debate](https://mlsanity.com/paper/2509.11035/) (2025)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions](https://mlsanity.com/paper/2511.11789/) (2025)

## Abstract (arXiv)

Multi-agent systems, which consist of multiple AI models interacting within a shared environment, are increasingly used for persona-based interactions. However, if not carefully designed, these systems can reinforce implicit biases in large language models (LLMs), raising concerns about fairness and equitable representation. We present MALIBU, a novel benchmark developed to assess the degree to which LLM-based multi-agent systems implicitly reinforce social biases and stereotypes. MALIBU evaluates bias in LLM-based multi-agent systems through scenario-based assessments. AI models complete tasks within predefined contexts, and their responses undergo evaluation by an LLM-based multi-agent judging system in two phases. In the first phase, judges score responses labeled with specific demographic personas (e.g., gender, race, religion) across four metrics. In the second phase, judges compare paired responses assigned to different personas, scoring them and selecting the superior response. Our study quantifies biases in LLM-generated outputs, revealing that bias mitigation may favor marginalized personas over true neutrality, emphasizing the need for nuanced detection, balanced fairness strategies, and transparent evaluation benchmarks in multi-agent systems.