mlsanityData to 7 Oct 2026

Model-Ensemble Trust-Region Policy Optimization

arXiv 1802.10592 ↗First posted 28 Feb 2018cs.LG cs.AI cs.ROcited by 214

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.

Key facts. Cited by 214 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 3 of its 45 references that are in the map. First posted on arXiv 28 Feb 2018.

Builds on

3 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug

Other references · 2

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 5; 2020: 7; 2021: 66; 2022: 44; 2023: 38; 2024: 28; 2025: 13; 2026: 10.

214 papers.

Sort:

By area:

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2020Mastering Atari with Discrete World Modelscited by 990
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  11. 2022Masked World Models for Visual Controlcited by 169
  12. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
38 more
  1. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  3. 2021Learning to drive from a world on railscited by 114
  4. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  5. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  6. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  7. 2019Exploring Model-based Planning with Policy Networkscited by 90
  8. 2021Gradients are Not All You Needcited by 82
  9. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  10. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  11. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  12. 2022A Survey on Model-based Reinforcement Learningcited by 65
  13. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  14. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  15. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  16. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  17. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  18. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  19. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  20. 2021Vector Quantized Models for Planningcited by 47
  21. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  22. 2023A Survey on Causal Reinforcement Learningcited by 43
  23. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  24. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  25. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  26. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  27. 2021Adaptive-Control-Oriented Meta-Learning for Nonlinear Systemscited by 38
  28. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  29. 2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
  30. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  31. 2021Neural Auction: End-to-End Learning of Auction Mechanisms for E-Commerce Advertisingcited by 31
  32. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
  33. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  34. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  35. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  36. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  37. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  38. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 25

50 most cited of 214 All 214 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning
  2. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  3. 2020Adversarial Imitation Learning via Random Search
  4. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
  5. 2024PWM: Policy Learning with Multi-Task World Models
  6. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RL
  7. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  8. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model