mlsanityData to 14 Oct 2026

Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic

arXiv 2112.10504 ↗First posted 16 Dec 2021cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning algorithms, which aim to learn a model of the environment to make decisions, are more sample efficient than their model-free counterparts. The sample efficiency of model-based approaches relies on whether the model can well approximate the environment. However, learning an accurate model is challenging, especially in complex and noisy environments. To tackle this problem, we propose the conservative model-based actor-critic (CMBAC), a novel approach that achieves high sample efficiency without the strong reliance on accurate learned models. Specifically, CMBAC learns multiple estimates of the Q-value function from a set of inaccurate models and uses the average of the bottom-k estimates -- a conservative estimate -- to optimize the policy. An appealing feature of CMBAC is that the conservative estimates effectively encourage the agent to avoid unreliable "promising actions" -- whose values are high in only a small fraction of the models. Experiments demonstrate that CMBAC significantly outperforms state-of-the-art approaches in terms of sample efficiency on several challenging tasks, and the proposed method is more robust than previous methods in noisy environments.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

30 of 49 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2016OpenAI Gymcited by 1,736
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
18 more
  1. 2016Deep Exploration via Bootstrapped DQNcited by 527
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  3. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  5. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  6. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  8. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  9. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  10. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  11. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  12. 2019Exploring Model-based Planning with Policy Networkscited by 90
  13. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  14. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  15. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  16. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  17. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
  18. 2019Uncertainty-aware Model-based Policy Optimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 7; 2025: 4; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  2. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 18
  3. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  4. 2025Apollo-MILP: An Alternating Prediction-Correction Neural Solving Framework for Mixed-Integer Linear Programmingcited by 16
  5. 2022Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributionscited by 15
  6. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  7. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  8. 2024Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designscited by 9
  9. 2024Learning to Stop Cut Generation for Efficient Mixed-Integer Linear Programmingcited by 8
  10. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  11. 2023State Sequences Prediction via Fourier Transform for Representation Learningcited by 7
  12. 2023A Circuit Domain Generalization Framework for Efficient Logic Synthesis in Chip Designcited by 6
11 more
  1. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  2. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 5
  3. 2023Generalization in Visual Reinforcement Learning with the Reward Sequence Distributioncited by 4
  4. 2025A Survey of Reinforcement Learning for Optimization in Automationcited by 4
  5. 2022NARS vs. Reinforcement learning: ONA vs. Q-Learningcited by 1
  6. 2024STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processingcited by 1
  7. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  8. 2024A Case for Validation Buffer in Pessimistic Actor-Criticcited by 0
  9. 2025FM-EAC: Feature Model-based Enhanced Actor-Critic for Multi-Task Control in Dynamic Environmentscited by 0
  10. 2026FS-I2P:A Hierarchical Focus-Sweep Registration Network with Dynamically Allocated Depthcited by 0
  11. 2026Mask 2D-3D: Adaptive Dual-Masked Autoencoder Network for Image-to-Point Cloud Registrationcited by 0