When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Reinforcement learning (RL) algorithms can be divided into two classes: model-free algorithms, which are sample-inefficient, and model-based algorithms, which suffer from model bias. Dyna-style algorithms combine these two approaches by using simulated data from an estimated environmental model to accelerate model-free training. However, their efficiency is compromised when the estimated model is inaccurate. Previous works address this issue by using model ensembles or pretraining the estimated model with data collected from the real environment, increasing computational and sample complexity. To tackle this issue, we introduce an out-of-distribution (OOD) data filter that removes simulated data from the estimated model that significantly diverges from data collected in the real environment. We show theoretically that this technique enhances the quality of simulated data. With the help of the OOD data filter, the data simulated from the estimated model better mimics the data collected by interacting with the real model. This improvement is evident in the critic updates compared to using the simulated data without the OOD data filter. Our experiment integrates the data filter into the model-based policy optimization (MBPO) algorithm. The results demonstrate that our method requires fewer interactions with the real environment to achieve a higher level of optimality than MBPO, even without a model ensemble.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 35 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
20 of 35 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
- 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 857
- 2021Generalized Out-of-Distribution Detection: A Surveycited by 799
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 756
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 754
- 2021Towards Out-Of-Distribution Generalization: A Surveycited by 432
- 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
- 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 338
- 2017Deep Reinforcement Learning: An Overviewcited by 327
- 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
- 2018Residual Reinforcement Learning for Robot Controlcited by 244
7 more
- 2018Model-Ensemble Trust-Region Policy Optimizationcited by 216
- 2019Benchmarking Model-Based Reinforcement Learningcited by 158
- 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
- 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
- 2015Learning Contact-Rich Manipulation Skills with Guided Policy Searchcited by 37
- 2019Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learningcited by 15
- 2022Accelerating Model-Free Policy Optimization Using Model-Based Gradient: A Composite Optimization Perspectivecited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.