# Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning arXiv: https://arxiv.org/abs/2601.07463 ยท HTML: https://mlsanity.com/paper/2601.07463/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. - Builds on 32 of its 57 references that are in the map. - First posted on arXiv 12 Jan 2026. ## Placed in - World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/) - Offline RL and imitation learning from data / [Offline multi-agent RL](https://mlsanity.com/offlinerl/offline-multi-agent-rl/) ## Builds on (most cited first, 30 of 32) - [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020) - [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017) - [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023) - [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020) - [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017) - [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020) - [Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage](https://mlsanity.com/paper/2107.06226/) (2021) - [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021) - [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023) - [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021) - [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021) - [Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization](https://mlsanity.com/paper/2307.11620/) (2023) - [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021) - [Model-based Adversarial Meta-Reinforcement Learning](https://mlsanity.com/paper/2006.08875/) (2020) - [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024) - [A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem](https://mlsanity.com/paper/2305.17198/) (2023) - [Model-based Offline Reinforcement Learning with Count-based Conservatism](https://mlsanity.com/paper/2307.11352/) (2023) - [Model-Based Offline Meta-Reinforcement Learning with Regularization](https://mlsanity.com/paper/2202.02929/) (2022) - [Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization](https://mlsanity.com/paper/2412.07639/) (2024) - [Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation](https://mlsanity.com/paper/2203.07322/) (2022) - [Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL](https://mlsanity.com/paper/2209.09845/) (2022) - [AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2311.02194/) (2023) - [Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL](https://mlsanity.com/paper/2402.05724/) (2024) - [Efficient Multi-agent Reinforcement Learning by Planning](https://mlsanity.com/paper/2405.11778/) (2024) - [Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation](https://mlsanity.com/paper/2406.09068/) (2024) - [ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization](https://mlsanity.com/paper/2410.01954/) (2024) - [Multi-Agent Reinforcement Learning in Wireless Distributed Networks for 6G](https://mlsanity.com/paper/2502.05812/) (2025) - [MA-Dreamer: Coordination and communication through shared imagination](https://mlsanity.com/paper/2204.04687/) (2022) - [MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment](https://mlsanity.com/paper/2312.03644/) (2023) - [Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank](https://mlsanity.com/paper/2410.01101/) (2024) ## Built on by (most cited first, 2 of 2) - [MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mind](https://mlsanity.com/paper/2603.00808/) (2026) - [MA-WAM: Multi-Agent World-Action Model for Test-Time Planning](https://mlsanity.com/paper/2609.31281/) (2026) ## Abstract (arXiv) Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within the dataset distribution, resulting in overly conservative policies that struggle to generalize beyond the support of the data. While model-based approaches offer a promising solution by expanding the original dataset with synthetic data generated from a learned world model, the high dimensionality, non-stationarity, and complexity of multi-agent systems make it challenging to accurately estimate the transitions and reward functions in offline MARL. Given the difficulty of directly modeling joint dynamics, we propose a local-to-global (LOGO) world model, a novel framework that leverages local predictions-which are easier to estimate-to infer global state dynamics, thus improving prediction accuracy while implicitly capturing agent-wise dependencies. Using the trained world model, we generate synthetic data to augment the original dataset, expanding the effective state-action space. To ensure reliable policy learning, we further introduce an uncertainty-aware sampling mechanism that adaptively weights synthetic data by prediction uncertainty, reducing approximation error propagation to policies. In contrast to conventional ensemble-based methods, our approach requires only an additional encoder for uncertainty estimation, significantly reducing computational overhead while maintaining accuracy. Extensive experiments across 8 scenarios against 8 baselines demonstrate that our method surpasses state-of-the-art baselines on standard offline MARL benchmarks, establishing a new model-based baseline for generalizable offline multi-agent learning.