Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
Not on the map. Why?
Abstract (arXiv)
In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL shows the prospect of being more sample-efficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 26 Mar 2021.
Builds on
31 of 40 references.
References · 31
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2015Trust Region Policy Optimizationcited by 2,503
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2017Deep Reinforcement Learning that Matterscited by 727
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
19 more
- 2019Challenges of Real-World Reinforcement Learningcited by 282
- 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
- 2019Benchmarking Model-Based Reinforcement Learningcited by 158
- 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
- 2016GPflow: A Gaussian process library using TensorFlowcited by 139
- 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
- 2019Exploring Model-based Planning with Policy Networkscited by 90
- 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
- 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
- 2020A Framework for Interdomain and Multioutput Gaussian Processescited by 47
- 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 45
- 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
- 2020A Tutorial on Sparse Gaussian Processes and Variational Inferencecited by 29
- 2019Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learningcited by 18
- 2019A Unified Bellman Optimality Principle Combining Reward Maximization and Empowermentcited by 15
- 2018Boosting Trust Region Policy Optimization by Normalizing Flows Policycited by 11
- 2016Planning with Information-Processing Constraints and Model Uncertainty in Markov Decision Processescited by 7
- 2018Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networkscited by 3
Built on by
Citing papers per year: 2021: 1; 2022: 1.
2 papers.