# Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion

arXiv: https://arxiv.org/abs/1912.05328 · HTML: https://mlsanity.com/paper/1912.05328/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 13 of 13)

- [Learning to run a Power Network Challenge: a Retrospective Analysis](https://mlsanity.com/paper/2103.03104/) (2021)
- [Model-Based Uncertainty in Value Functions](https://mlsanity.com/paper/2302.12526/) (2023)
- [OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion](https://mlsanity.com/paper/2112.06061/) (2021)
- [Reinforcement learning for Energies of the future and carbon neutrality: a Challenge Design](https://mlsanity.com/paper/2207.10330/) (2022)
- [Measuring and modeling the motor system with machine learning](https://mlsanity.com/paper/2103.11775/) (2021)
- [Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.09418/) (2022)
- [Value-Distributional Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.06590/) (2023)
- [Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization](https://mlsanity.com/paper/2312.04386/) (2023)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion](https://mlsanity.com/paper/2407.11658/) (2024)
- [A study of first-passage time minimization via Q-learning in heated gridworlds](https://mlsanity.com/paper/2110.02129/) (2021)
- [Applications of Reinforcement Learning in Deregulated Power Market: A Comprehensive Review](https://mlsanity.com/paper/2205.08369/) (2022)
- [Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications](https://mlsanity.com/paper/2412.01839/) (2024)

## Abstract (arXiv)

By integrating dynamics models into model-free reinforcement learning (RL) methods, model-based value expansion (MVE) algorithms have shown a significant advantage in sample efficiency as well as value estimation. However, these methods suffer from higher function approximation errors than model-free methods in stochastic environments due to a lack of modeling the environmental randomness. As a result, their performance lags behind the best model-free algorithms in some challenging scenarios. In this paper, we propose a novel Hybrid-RL method that builds on MVE, namely the Risk Averse Value Expansion (RAVE). With imaginative rollouts generated by an ensemble of probabilistic dynamics models, we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on a range of challenging environments show that by modeling the uncertainty completely, RAVE substantially enhances the robustness of previous model-based methods, and yields state-of-the-art performance. With this technique, our solution gets the first place in NeurIPS 2019: Learn to Move.