mlsanityData to 14 Oct 2026

Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression

arXiv 2511.11973 ↗First posted 15 Nov 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme Q-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient β via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 38 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

28 of 38 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios25 Mar
  2. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  3. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  4. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  5. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning28 Jan
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  9. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
2 more
  1. 2018Learning to Navigate in Cities Without a Mapcited by 104
  2. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy
  2. 2024Stabilizing Extreme Q-learning by Maclaurin Expansion
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  4. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  5. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  6. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  7. 2024Strategically Conservative Q-Learning
  8. 2024Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning