mlsanityData to 7 Oct 2026

Bootstrapping Expectiles in Reinforcement Learning

arXiv 2406.04081 ↗First posted 6 Jun 2024cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Many classic Reinforcement Learning (RL) algorithms rely on a Bellman operator, which involves an expectation over the next states, leading to the concept of bootstrapping. To introduce a form of pessimism, we propose to replace this expectation with an expectile. In practice, this can be very simply done by replacing the L2 loss with a more general expectile loss for the critic. Introducing pessimism in RL is desirable for various reasons, such as tackling the overestimation problem (for which classic solutions are double Q-learning or the twin-critic approach of TD3) or robust RL (where transitions are adversarial). We study empirically these two cases. For the overestimation problem, we show that the proposed approach, ExpectRL, provides better results than a classic twin-critic. On robust RL benchmarks, involving changes of the environment, we show that our approach is more robust than classic RL algorithms. We also introduce a variation of ExpectRL combined with domain randomization which is competitive with state-of-the-art robust RL agents. Eventually, we also extend ExpectRL with a mechanism for choosing automatically the expectile value, that is the degree of pessimism

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

33 of 50 references.

References · 33

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2017Robust Adversarial Reinforcement Learningcited by 439
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  9. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  10. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  11. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  12. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
21 more
  1. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  2. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  3. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  4. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  5. 2019Risk Averse Robust Adversarial Reinforcement Learningcited by 54
  6. 2019Wasserstein Robust Reinforcement Learningcited by 50
  7. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  8. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  9. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  10. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  11. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  12. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 39
  13. 2021Regularized Policies are Reward Robustcited by 32
  14. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  15. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  16. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  17. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  18. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  19. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 15
  20. 2022Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecificationcited by 11
  21. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reinforcement learning with an expectile-based objectivecited by 0