# YOLO-MARL: You Only LLM Once for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2410.03997 · HTML: https://mlsanity.com/paper/2410.03997/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 5 Oct 2024.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 32)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://mlsanity.com/paper/2310.10701/) (2023)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023)
- [MindAgent: Emergent Gaming Interaction](https://mlsanity.com/paper/2309.09971/) (2023)
- [Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach](https://mlsanity.com/paper/2312.11865/) (2023)
- [LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions](https://mlsanity.com/paper/2405.11106/) (2024)
- [Enhance Reasoning for Large Language Models in the Game Werewolf](https://mlsanity.com/paper/2402.02330/) (2024)
- [Graph-enhanced Large Language Models in Asynchronous Plan Reasoning](https://mlsanity.com/paper/2402.02805/) (2024)
- [Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems](https://mlsanity.com/paper/2307.06187/) (2023)
- [Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yet](https://mlsanity.com/paper/2401.03630/) (2024)

## Built on by (most cited first, 9 of 9)

- [Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs](https://mlsanity.com/paper/2511.06134/) (2025)
- [Multi-Agent Systems for Robotic Autonomy with LLMs](https://mlsanity.com/paper/2505.05762/) (2025)
- [Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning](https://mlsanity.com/paper/2603.07972/) (2026)
- [RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarms](https://mlsanity.com/paper/2507.01378/) (2025)
- [AURA: Adaptive Unified Reasoning and Automation with LLM-Guided MARL for NextG Cellular Networks](https://mlsanity.com/paper/2511.17506/) (2025)
- [Proof2Silicon: Prompt Repair for Verified Code and Hardware Generation via Reinforcement Learning](https://mlsanity.com/paper/2509.06239/) (2025)
- [Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automation](https://mlsanity.com/paper/2509.13368/) (2025)
- [A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturing](https://mlsanity.com/paper/2608.07148/) (2026)
- [A Schema Bounded Language Model for Refining Robot Policies Without Destabilizing Local Learning](https://mlsanity.com/paper/2609.05133/) (2026)

## Abstract (arXiv)

Advancements in deep multi-agent reinforcement learning (MARL) have positioned it as a promising approach for decision-making in cooperative games. However, it still remains challenging for MARL agents to learn cooperative strategies for some game environments. Recently, large language models (LLMs) have demonstrated emergent reasoning capabilities, making them promising candidates for enhancing coordination among the agents. However, due to the model size of LLMs, it can be expensive to frequently infer LLMs for actions that agents can take. In this work, we propose You Only LLM Once for MARL (YOLO-MARL), a novel framework that leverages the high-level task planning capabilities of LLMs to improve the policy learning process of multi-agents in cooperative games. Notably, for each game environment, YOLO-MARL only requires one time interaction with LLMs in the proposed strategy generation, state interpretation and planning function generation modules, before the MARL policy training process. This avoids the ongoing costs and computational time associated with frequent LLMs API calls during training. Moreover, trained decentralized policies based on normal-sized neural networks operate independently of the LLM. We evaluate our method across two different environments and demonstrate that YOLO-MARL outperforms traditional MARL algorithms.