mlsanityData to 7 Oct 2026

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

arXiv 2601.09667 ↗First posted 14 Jan 2026cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce Multi-Agent Test-Time Reinforcement Learning (MATTRL), a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67% over a multi-agent baseline, and by 8.67% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 20 of its 28 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

20 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning12 Mar
  2. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb

Other references · 18

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  5. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  7. 2025Humanity's Last Examcited by 467
  8. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  9. 2024The Faiss librarycited by 370
  10. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  11. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  12. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
6 more
  1. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 67
  2. 2025Test-Time Learning for Large Language Modelscited by 46
  3. 2025Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignmentcited by 17
  4. 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Modelscited by 17
  5. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 11
  6. 2025How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparisoncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Unifying Temporal and Structural Credit Assignment in LLM-Based Multi-Agent Prompt Optimizationcited by 2
  2. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 2
  3. 2026Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discoverycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025MarsRL: Advancing Multi-Agent Reasoning System via Reinforcement Learning with Agentic Pipeline Parallelism
  2. 2026Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
  3. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learning
  4. 2025DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs
  5. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
  6. 2025OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning
  7. 2026MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation
  8. 2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signals