mlsanityData to 7 Oct 2026

Extreme Q-Learning: MaxEnt RL without Entropy

arXiv 2301.02328 ↗First posted 5 Jan 2023cs.LG cs.AI cs.ROcited by 113

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our Extreme Q-Learning framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by 10+ points on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 53 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

29 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021Offline RL Without Off-Policy Evaluation16 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2016Layer Normalizationcited by 5,579
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2018DeepMind Control Suitecited by 935
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
11 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2019On Variational Bounds of Mutual Informationcited by 550
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  6. 2017A unified view of entropy-regularized Markov decision processescited by 174
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  8. 2020Logistic Q-Learningcited by 34
  9. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  10. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  11. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 25; 2025: 37; 2026: 27.

113 papers.

Sort:

By area:

  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  2. 2025Flow Q-Learningcited by 162
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  4. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  5. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  6. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  8. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  9. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  10. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  11. 2025Horizon Reduction Makes RL Scalablecited by 55
  12. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
38 more
  1. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  2. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  3. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  4. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  5. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  6. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  7. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  8. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  9. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  10. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  11. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  12. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  13. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  14. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  15. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  16. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  17. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  18. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
  19. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  20. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learningcited by 16
  21. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 15
  22. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  23. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  24. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  25. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  26. 2024Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networkscited by 9
  27. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  28. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  29. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  30. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  31. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  32. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  33. 2024Value Improved Actor Critic Algorithmscited by 6
  34. 2024Random Latent Exploration for Deep Reinforcement Learningcited by 6
  35. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  36. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  37. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  38. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
  3. 2020Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
  4. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
  5. 2025Failure Modes of Maximum Entropy RLHF
  6. 2025Maximum Entropy Reinforcement Learning with Diffusion Policy
  7. 2024Stabilizing Extreme Q-learning by Maclaurin Expansion
  8. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learning