# Models as Agents: Optimizing Multi-Step Predictions of Interactive Local Models in Model-Based Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.17984 · HTML: https://mlsanity.com/paper/2303.17984/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2023.

## Built on by (most cited first, 10 of 10)

- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective](https://mlsanity.com/paper/2505.20922/) (2025)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Model](https://mlsanity.com/paper/2309.04615/) (2023)
- [Reflective Policy Optimization](https://mlsanity.com/paper/2406.03678/) (2024)
- [Continuum-Interaction-Driven Intelligence: Human-Aligned Neural Architecture via Crystallized Reasoning and Fluid Generation](https://mlsanity.com/paper/2504.09301/) (2025)
- [Multi-agent Uncertainty-Aware Pessimistic Model-Based Reinforcement Learning for Connected Autonomous Vehicles](https://mlsanity.com/paper/2503.20462/) (2025)
- [Joint Trajectory, RIS, and Computation Offloading Optimization via Decentralized Model-Based PPO in Urban Multi-UAV Mobile Edge Computing](https://mlsanity.com/paper/2603.20238/) (2026)
- [Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2607.19809/) (2026)
- [Waggle: Learning One Anonymous Local Law for Self-Organizing LLM Swarms](https://mlsanity.com/paper/2609.34136/) (2026)

## Abstract (arXiv)

Research in model-based reinforcement learning has made significant progress in recent years. Compared to single-agent settings, the exponential dimension growth of the joint state-action space in multi-agent systems dramatically increases the complexity of the environment dynamics, which makes it infeasible to learn an accurate global model and thus necessitates the use of agent-wise local models. However, during multi-step model rollouts, the prediction of one local model can affect the predictions of other local models in the next step. As a result, local prediction errors can be propagated to other localities and eventually give rise to considerably large global errors. Furthermore, since the models are generally used to predict for multiple steps, simply minimizing one-step prediction errors regardless of their long-term effect on other models may further aggravate the propagation of local errors. To this end, we propose Models as AGents (MAG), a multi-agent model optimization framework that reversely treats the local models as multi-step decision making agents and the current policies as the dynamics during the model rollout process. In this way, the local models are able to consider the multi-step mutual affect between each other before making predictions. Theoretically, we show that the objective of MAG is approximately equivalent to maximizing a lower bound of the true environment return. Experiments on the challenging StarCraft II benchmark demonstrate the effectiveness of MAG.