# LangMARL: Natural Language Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.00722 · HTML: https://mlsanity.com/paper/2604.00722/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 20 of 20)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Agents: An Open-source Framework for Autonomous Language Agents](https://mlsanity.com/paper/2309.07870/) (2023)
- [Symbolic Learning Enables Self-Evolving Agents](https://mlsanity.com/paper/2406.18532/) (2024)
- [SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning](https://mlsanity.com/paper/2502.04780/) (2025)
- [Self-Evolving Multi-Agent Systems via Textual Backpropagation](https://mlsanity.com/paper/2506.09046/) (2025)
- [CoMAL: Collaborative Multi-Agent Large Language Models for Mixed-Autonomy Traffic](https://mlsanity.com/paper/2410.14368/) (2024)
- [SOCIA-nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation](https://mlsanity.com/paper/2505.12006/) (2025)

## Built on by (most cited first, 4 of 4)

- [Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces](https://mlsanity.com/paper/2605.02801/) (2026)
- [Scaling Multi-agent Systems: A Smart Middleware for Improving Agent Interactions](https://mlsanity.com/paper/2604.03430/) (2026)
- [Information-Directed Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2605.29405/) (2026)
- [ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failures](https://mlsanity.com/paper/2609.16986/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.