mlsanityData to 14 Oct 2026

M3PO: Massively Multi-Task Model-Based Policy Optimization

arXiv 2506.21782 ↗First posted 26 Jun 2025cs.LG cs.ROcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We introduce Massively Multi-Task Model-Based Policy Optimization (M3PO), a scalable model-based reinforcement learning (MBRL) framework designed to address sample inefficiency in single-task settings and poor generalization in multi-task domains. Existing model-based approaches like DreamerV3 rely on pixel-level generative models that neglect control-centric representations, while model-free methods such as PPO suffer from high sample complexity and weak exploration. M3PO integrates an implicit world model, trained to predict task outcomes without observation reconstruction, with a hybrid exploration strategy that combines model-based planning and model-free uncertainty-driven bonuses. This eliminates the bias-variance trade-off in prior methods by using discrepancies between model-based and model-free value estimates to guide exploration, while maintaining stable policy updates through a trust-region optimizer. M3PO provides an efficient and robust alternative to existing model-based policy optimization approaches and achieves state-of-the-art performance across multiple benchmarks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 24 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

19 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Model-based Policy Optimization using Symbolic World Model18 Jul

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2020Mastering Atari with Discrete World Modelscited by 1,007
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
6 more
  1. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  2. 2018Distributed Prioritized Experience Replaycited by 269
  3. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  4. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  5. 2016DeepMind Labcited by 74
  6. 2018Policy Optimization with Model-based Explorationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learningcited by 0
  2. 2026Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
  2. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  3. 2020Ready Policy One: World Building Through Active Learning
  4. 2024PWM: Policy Learning with Multi-Task World Models
  5. 2020A Game Theoretic Framework for Model Based Reinforcement Learning
  6. 2025Look Before Leap: Look-Ahead Planning with Uncertainty in Reinforcement Learning
  7. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  8. 2026TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing