# Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

arXiv: https://arxiv.org/abs/2601.09667 · HTML: https://mlsanity.com/paper/2601.09667/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 20 of 20)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Scaling Large Language Model-based Multi-Agent Collaboration](https://mlsanity.com/paper/2406.07155/) (2024)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks](https://mlsanity.com/paper/2411.04468/) (2024)
- [MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2502.18439/) (2025)
- [SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines](https://mlsanity.com/paper/2502.14739/) (2025)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [Test-Time Learning for Large Language Models](https://mlsanity.com/paper/2505.20633/) (2025)
- [Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment](https://mlsanity.com/paper/2502.16863/) (2025)
- [Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models](https://mlsanity.com/paper/2505.10554/) (2025)
- [Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agents](https://mlsanity.com/paper/2506.07388/) (2025)
- [How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparison](https://mlsanity.com/paper/2506.14448/) (2025)

## Built on by (most cited first, 3 of 3)

- [AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?](https://mlsanity.com/paper/2608.00155/) (2026)
- [Unifying Temporal and Structural Credit Assignment in LLM-Based Multi-Agent Prompt Optimization](https://mlsanity.com/paper/2605.30227/) (2026)
- [Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery](https://mlsanity.com/paper/2606.08728/) (2026)

## Abstract (arXiv)

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce Multi-Agent Test-Time Reinforcement Learning (MATTRL), a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67% over a multi-agent baseline, and by 8.67% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.