# ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration

arXiv: https://arxiv.org/abs/2411.00053 · HTML: https://mlsanity.com/paper/2411.00053/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 49)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [More Agents Is All You Need](https://mlsanity.com/paper/2402.05120/) (2024)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?](https://mlsanity.com/paper/2402.18272/) (2024)

## Built on by (most cited first, 24 of 24)

- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate](https://mlsanity.com/paper/2509.05396/) (2025)
- [CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards](https://mlsanity.com/paper/2510.08529/) (2025)
- [Multi-Agent Debate for LLM Judges with Adaptive Stability Detection](https://mlsanity.com/paper/2510.12697/) (2025)
- [How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs](https://mlsanity.com/paper/2507.08960/) (2025)
- [DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoning](https://mlsanity.com/paper/2505.15734/) (2025)
- [Interactive Learning for LLM Reasoning](https://mlsanity.com/paper/2509.26306/) (2025)
- [DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning](https://mlsanity.com/paper/2512.07132/) (2025)
- [Emergent Bias and Fairness in Multi-Agent Decision Systems](https://mlsanity.com/paper/2512.16433/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [MarsRL: Advancing Multi-Agent Reasoning System via Reinforcement Learning with Agentic Pipeline Parallelism](https://mlsanity.com/paper/2511.11373/) (2025)
- [Asymmetric Actor-Critic for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.00304/) (2026)
- [Multi-Agent Dialectical Refinement for Enhanced Argument Classification](https://mlsanity.com/paper/2603.27451/) (2026)
- [Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents](https://mlsanity.com/paper/2604.05808/) (2026)
- [AgentNLQ: A General-Purpose Agent for Natural Language to SQL](https://mlsanity.com/paper/2605.19010/) (2026)
- [From Argument Components to Graphs: A Multi-Agent Debate with Confidence Gating for Argument Relations](https://mlsanity.com/paper/2606.16047/) (2026)
- [Debating for Better Reasoning: An Unsupervised Multimodal Approach](https://mlsanity.com/paper/2505.14627/) (2025)
- [When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoning](https://mlsanity.com/paper/2605.06772/) (2026)
- [Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs](https://mlsanity.com/paper/2605.08686/) (2026)
- [Examining Agents' Bias Amplification versus Suppression in Multi-Agent Systems](https://mlsanity.com/paper/2605.28098/) (2026)
- [CAF-Gen: A Multi-Agent System for Enriching Argumentation Structures](https://mlsanity.com/paper/2606.06646/) (2026)
- [SEPAL: Separated Expert Pairs with Answer-Level Fusion for Reliable LLM Collaboration](https://mlsanity.com/paper/2609.39645/) (2026)
- [Observe, Share, or Predict? Disentangling Partner Information in Embodied Multi-Agent Collaboration](https://mlsanity.com/paper/2610.16895/) (2026)
- [Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning](https://mlsanity.com/paper/2610.17456/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated a remarkable ability to serve as general-purpose tools for various language-based tasks. Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models. While these paradigms show promise in improving model efficacy, most works in this area treat collaboration as an emergent behavior, rather than a learned behavior. In doing so, current multi-agent frameworks rely on collaborative behaviors to have been sufficiently trained into off-the-shelf models. To address this limitation, we propose ACC-Collab, an Actor-Critic based learning framework to produce a two-agent team (an actor-agent and a critic-agent) specialized in collaboration. We demonstrate that ACC-Collab outperforms SotA multi-agent techniques on a wide array of benchmarks.