# Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.08174 · HTML: https://mlsanity.com/paper/2604.08174/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Mean Flows for One-step Generative Modeling](https://mlsanity.com/paper/2505.13447/) (2025)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Flow Q-Learning](https://mlsanity.com/paper/2502.02538/) (2025)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Efficient Diffusion Policies for Offline Reinforcement Learning](https://mlsanity.com/paper/2305.20081/) (2023)
- [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021)
- [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [Diffusion Guidance Is a Controllable Policy Improvement Operator](https://mlsanity.com/paper/2505.23458/) (2025)
- [Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization](https://mlsanity.com/paper/2307.11620/) (2023)
- [Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL](https://mlsanity.com/paper/2307.01472/) (2023)
- [Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization](https://mlsanity.com/paper/2412.07639/) (2024)
- [AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2311.02194/) (2023)
- [Multi-agent Coordination via Flow Matching](https://mlsanity.com/paper/2511.05005/) (2025)
- [Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.00521/) (2023)
- [OM2P: Offline Multi-Agent Mean-Flow Policy](https://mlsanity.com/paper/2508.06269/) (2025)

## Built on by (most cited first, 3 of 3)

- [MA-WAM: Multi-Agent World-Action Model for Test-Time Planning](https://mlsanity.com/paper/2609.31281/) (2026)
- [G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies](https://mlsanity.com/paper/2609.31286/) (2026)
- [Multi-Agent Coordination via Support-Preserving Distillation](https://mlsanity.com/paper/2610.10087/) (2026)

## Abstract (arXiv)

Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM2P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM2P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM2P efficiently achieves performance comparable to state-of-the-art methods.