mlsanityData to 14 Oct 2026

Robust Offline Reinforcement learning with Heavy-Tailed Rewards

arXiv 2310.18715 ↗First posted 28 Oct 2023cs.LG cs.AI stat.MLcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper endeavors to augment the robustness of offline reinforcement learning (RL) in scenarios laden with heavy-tailed rewards, a prevalent circumstance in real-world applications. We propose two algorithmic frameworks, ROAM and ROOM, for robust off-policy evaluation and offline policy optimization (OPO), respectively. Central to our frameworks is the strategic incorporation of the median-of-means method with offline RL, enabling straightforward uncertainty estimation for the value function estimator. This not only adheres to the principle of pessimism in OPO but also adeptly manages heavy-tailed rewards. Theoretical results and extensive experiments demonstrate that our two frameworks outperform existing methods on the logged dataset exhibits heavy-tailed reward distributions. The implementation of the proposal is available at https://github.com/Mamba413/ROOM.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 90 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

56 of 90 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Bi-Level Offline Policy Optimization with Limited Exploration10 Oct
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  4. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes18 Sep
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning10 Sep
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  11. 2021Offline RL Without Off-Policy Evaluation16 Jun
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
11 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inference6 Feb
  6. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  11. 2019Mean estimation and regression under heavy-tailed distributions--a surveycited by 136
  12. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
21 more
  1. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  2. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  3. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  4. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  5. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  6. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  7. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  8. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  9. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  10. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  11. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  12. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  13. 2018Almost Optimal Algorithms for Linear Stochastic Bandits with Heavy-Tailed Payoffscited by 31
  14. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  15. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  16. 2021No-Regret Reinforcement Learning with Heavy-Tailed Rewardscited by 14
  17. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  18. 2021Breaking the Moments Condition Barrier: No-Regret Algorithm for Bandits with Super Heavy-Tailed Payoffscited by 8
  19. 2022Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approachcited by 7
  20. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  21. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2026Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noisecited by 2
  2. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  3. 2025Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Ratescited by 1
  4. 2026Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noisecited by 1
  5. 2026Best-of-Both-Worlds for Heavy-Tailed Markov Decision Processescited by 1
  6. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  7. 2026DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  2. 2023Survival Instinct in Offline Reinforcement Learning
  3. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  4. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
  5. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  6. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  7. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  8. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning