mlsanityData to 7 Oct 2026

MALT: Improving Reasoning with Multi-Agent LLM Training

arXiv 2412.01928 ↗First posted 2 Dec 2024cs.LG cs.AIcited by 71

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 36 of its 50 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

36 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents13 Aug
  2. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul
  3. 2024LLM Critics Help Catch LLM Bugs28 Jun

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
21 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  9. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  10. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
  13. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  14. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  15. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  16. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  17. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  18. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  19. 2024Understanding User Experience in Large Language Model Interactionscited by 43
  20. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  21. 2022Quasi-supervised Learning for Super-resolution PETcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 35; 2026: 36.

71 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  3. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  4. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  5. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  6. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  7. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  8. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  9. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  10. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  11. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  12. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoningcited by 20
38 more
  1. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  2. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  3. 2025Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learningcited by 18
  4. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 17
  5. 2026Colosseum: Auditing Collusion in Cooperative Multi-Agent Systemscited by 17
  6. 2025Learning from Peers in Reasoning Modelscited by 16
  7. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  8. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  9. 2025ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflectioncited by 11
  10. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  11. 2025A Survey on Parallel Reasoningcited by 11
  12. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  13. 2026Recursive Multi-Agent Systemscited by 10
  14. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  15. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoningcited by 8
  16. 2025Multi-Actor Generative Artificial Intelligence as a Game Enginecited by 8
  17. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  18. 2025AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generationcited by 7
  19. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  20. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  21. 2026PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Trainingcited by 6
  22. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoningcited by 6
  23. 2026MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?cited by 6
  24. 2025A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directionscited by 4
  25. 2025Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Votingcited by 4
  26. 2025Interactive Learning for LLM Reasoningcited by 4
  27. 2025OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoningcited by 4
  28. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 4
  29. 2025Efficient Multi-Agent System Training with Data Influence-Oriented Tree Searchcited by 3
  30. 2025Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learningcited by 3
  31. 2025System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systemscited by 2
  32. 2026Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimizationcited by 2
  33. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
  34. 2026Multi-Agent Computer Usecited by 2
  35. 2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signalscited by 2
  36. 2025MAS2: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systemscited by 1
  37. 2026Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generationcited by 1
  38. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing the General Agent Capabilities of Low-Parameter LLMs through Tuning and Multi-Branch Reasoning
  2. 2025Distilling LLM Agent into Small Models with Retrieval and Code Tools
  3. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
  4. 2026Critic-Guided Heterogeneous Multi-Agent Reasoning for Reliable Mathematical Problem Solving
  5. 2025Adaptive Reasoning Executor: A Collaborative Agent System for Efficient Reasoning
  6. 2024Mars-PO: Multi-Agent Reasoning System Preference Optimization
  7. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
  8. 2025MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems