mlsanityData to 7 Oct 2026

Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts

arXiv 2105.03363 ↗First posted 7 May 2021cs.LG cs.AI cs.MAcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper investigates the model-based methods in multi-agent reinforcement learning (MARL). We specify the dynamics sample complexity and the opponent sample complexity in MARL, and conduct a theoretic analysis of return discrepancy upper bound. To reduce the upper bound with the intention of low sample complexity during the whole learning process, we propose a novel decentralized model-based MARL method, named Adaptive Opponent-wise Rollout Policy Optimization (AORPO). In AORPO, each agent builds its multi-agent environment model, consisting of a dynamics model and multiple opponent models, and trains its policy with the adaptive opponent-wise rollout. We further prove the theoretic convergence of AORPO under reasonable assumptions. Empirical experiments on competitive and cooperative tasks demonstrate that AORPO can achieve improved sample efficiency with comparable asymptotic performance over the compared MARL methods.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

25 of 43 references.

References · 25

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  8. 2018Is Q-learning Provably Efficient?cited by 485
  9. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  10. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  11. 2017Learning with Opponent-Learning Awarenesscited by 218
  12. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
13 more
  1. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  2. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  3. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  4. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  5. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  6. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  7. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
  8. 2019Multi-Agent Actor-Critic with Hierarchical Graph Attention Networkcited by 33
  9. 2018Multiagent Soft Q-Learningcited by 26
  10. 2019Multi-Agent Reinforcement Learning with Multi-Step Generative Modelscited by 18
  11. 2019Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learningcited by 17
  12. 2020Multi-agent Trajectory Prediction with Fuzzy Query Attentioncited by 16
  13. 2018Learning to Communicate Implicitly By Actionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 5; 2024: 6; 2025: 4; 2026: 2.

24 papers.

Sort:

By area:

  1. 2022A Survey on Model-based Reinforcement Learningcited by 65
  2. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  3. 2021Model-Based Opponent Modelingcited by 21
  4. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 18
  5. 2022Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospectscited by 17
  6. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  7. 2024Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Taskcited by 17
  8. 2022Interaction Pattern Disentangling for Multi-Agent Reinforcement Learningcited by 8
  9. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  10. 2022Scalable Model-based Policy Optimization for Decentralized Networked Systemscited by 6
  11. 2022Multi-Agent Reinforcement Learning with Shared Resources for Inventory Managementcited by 5
  12. 2024On Diffusion Models for Multi-Agent Partial Observability: Shared Attractors, Error Bounds, and Composite Flowcited by 5
12 more
  1. 2024Beyond Local Views: Global State Inference with Diffusion Models for Cooperative Multi-Agent Reinforcement Learningcited by 4
  2. 2024Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimationcited by 4
  3. 2023CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learningcited by 3
  4. 2023VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Modelcited by 3
  5. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  6. 2025Ego-centric Learning of Communicative World Models for Autonomous Drivingcited by 2
  7. 2023Model-Based Decentralized Policy Optimizationcited by 1
  8. 2025PMAT: Optimizing Action Generation Order in Multi-Agent Reinforcement Learningcited by 1
  9. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  10. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0
  11. 2026CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learningcited by 0
  12. 2026Automated Approach for Solving Infinite-state Polynomial Reachability Gamescited by 0