# MARFT: Multi-Agent Reinforcement Fine-Tuning

arXiv: https://arxiv.org/abs/2504.16129 · HTML: https://mlsanity.com/paper/2504.16129/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 66 of its 111 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 66)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)

## Built on by (most cited first, 30 of 52)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [In-the-Flow Agentic System Optimization for Effective Planning and Tool Use](https://mlsanity.com/paper/2510.05592/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards](https://mlsanity.com/paper/2510.08529/) (2025)
- [Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs](https://mlsanity.com/paper/2510.11062/) (2025)
- [Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2509.03817/) (2025)
- [Heterogeneous Agent Collaborative Reinforcement Learning](https://mlsanity.com/paper/2603.02604/) (2026)
- [Position: The Real Barrier to LLM Agent Usability is Agentic ROI](https://mlsanity.com/paper/2505.17767/) (2025)
- [How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs](https://mlsanity.com/paper/2507.08960/) (2025)
- [Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation](https://mlsanity.com/paper/2511.02303/) (2025)
- [Counterfactual Credit Policy Optimization for Multi-Agent Collaboration](https://mlsanity.com/paper/2603.21563/) (2026)
- [Exact Is Easier: Credit Assignment for Cooperative LLM Agents](https://mlsanity.com/paper/2603.06859/) (2026)
- [End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning](https://mlsanity.com/paper/2506.02718/) (2025)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.19524/) (2025)
- [Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2601.21742/) (2026)
- [SAGE: Multi-Agent Self-Evolution for LLM Reasoning](https://mlsanity.com/paper/2603.15255/) (2026)
- [Interactive Learning for LLM Reasoning](https://mlsanity.com/paper/2509.26306/) (2025)
- [Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoning](https://mlsanity.com/paper/2603.01221/) (2026)
- [Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces](https://mlsanity.com/paper/2605.02801/) (2026)
- [Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration](https://mlsanity.com/paper/2505.21471/) (2025)
- [SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control](https://mlsanity.com/paper/2508.20018/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.23870/) (2025)
- [MAS2: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systems](https://mlsanity.com/paper/2509.24323/) (2025)
- [MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation](https://mlsanity.com/paper/2602.07848/) (2026)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)
- [Scaling Multiagent Systems with Process Rewards](https://mlsanity.com/paper/2601.23228/) (2026)
- [Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.09578/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based Multi-Agent Systems (LaMAS) offer a promising approach to collaborative tasks, but applying reinforcement learning to these systems remains challenging because classical multi-agent reinforcement learning assumptions do not fully capture their dependency structures, heterogeneous roles, and autoregressive outputs. This article provides a tutorial-style overview of the transition from reinforcement learning to reinforcement fine-tuning in single-agent and multi-agent settings, highlighting the differences between classical MARL and LLM-based multi-agent optimization. We introduce Multi-Agent Reinforcement Fine-Tuning (MARFT) and formalize LaMAS optimization as a Flexible Markov Game (Flex-MG). The MARFT framework combines predecessor-conditioned policy optimization, a centralized critic, parameter-efficient adapters, and token-level credit assignment. Our token-level analysis clarifies how within-action discounting affects returns and establishes consistency with action-level returns when the token discount is one. Experiments on fixed two- to four-agent mathematical and coding workflows compare MARFT with and without policy parameter sharing against IPPO. Both variants report higher mean evaluation scores in all 24 benchmark-team-size settings, while the variant without parameter sharing shows the clearest advantage in final training reward. Role-specification ablations examine how prescribed roles affect training. We conclude with application perspectives and open challenges, including dynamic workflows, sample efficiency, and system-level reliability. Code: https://github.com/jwliao-ai/MARFT.