# From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos

arXiv: https://arxiv.org/abs/2511.02762 · HTML: https://mlsanity.com/paper/2511.02762/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.03052/) (2024)
- [Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics](https://mlsanity.com/paper/1910.01465/) (2019)
- [PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1911.00025/) (2019)
- [Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2309.12696/) (2023)
- [EXPO: Stable Reinforcement Learning with Expressive Policies](https://mlsanity.com/paper/2507.07986/) (2025)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Maximum Entropy Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.10715/) (2023)
- [Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL](https://mlsanity.com/paper/2307.01472/) (2023)
- [Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination](https://mlsanity.com/paper/2504.12714/) (2025)
- [Fast Teammate Adaptation in the Presence of Sudden Policy Change](https://mlsanity.com/paper/2305.05911/) (2023)
- [Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation](https://mlsanity.com/paper/2503.21200/) (2025)

## Built on by (most cited first, 1 of 1)

- [Multi-Agent Reinforcement Learning via Agent-Specific Preference](https://mlsanity.com/paper/2608.08604/) (2026)

## Abstract (arXiv)

Training a team of agents from scratch in multi-agent reinforcement learning (MARL) is highly inefficient, much like asking beginners to play a symphony together without first practicing solo. Existing methods, such as offline or transferable MARL, can ease this burden, but they still rely on costly multi-agent data, which often becomes the bottleneck. In contrast, solo experiences are far easier to obtain in many important scenarios, e.g., collaborative coding, household cooperation, and search-and-rescue. To unlock their potential, we propose Solo-to-Collaborative RL (SoCo), a framework that transfers solo knowledge into cooperative learning. SoCo first pretrains a shared solo policy from solo demonstrations, then adapts it for cooperation during multi-agent training through a policy fusion mechanism that combines an MoE-like gating selector and an action editor. Experiments across diverse cooperative tasks show that SoCo significantly boosts the training efficiency and performance of backbone algorithms. These results demonstrate that solo demonstrations provide a scalable and effective complement to multi-agent data, making cooperative learning more practical and broadly applicable.