mlsanityData to 7 Oct 2026

Distorted Distributional Policy Evaluation for Offline Reinforcement Learning

arXiv 2601.01917 ↗First posted 5 Jan 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

While Distributional Reinforcement Learning (DRL) methods have demonstrated strong performance in online settings, its success in offline scenarios remains limited. We hypothesize that a key limitation of existing offline DRL methods lies in their approach to uniformly underestimate return quantiles. This uniform pessimism can lead to overly conservative value estimates, ultimately hindering generalization and performance. To address this, we introduce a novel concept called quantile distortion, which enables non-uniform pessimism by adjusting the degree of conservatism based on the availability of supporting data. Our approach is grounded in theoretical analysis and empirically validated, demonstrating improved performance over uniform pessimism.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 18 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

13 of 18 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  2. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  3. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 9

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 50
  8. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
  9. 2018Concentration bounds for empirical conditional value-at-risk: The unbounded casecited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  4. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  5. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  6. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  7. 2025Semi-pessimistic Reinforcement Learning
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning