mlsanityData to 7 Oct 2026

Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning

arXiv 2407.00699 ↗First posted 30 Jun 2024cs.LG cs.AIcited by 17

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based offline reinforcement learning (RL) is a compelling approach that addresses the challenge of learning from limited, static data by generating imaginary trajectories using learned models. However, these approaches often struggle with inaccurate value estimation from model rollouts. In this paper, we introduce a novel model-based offline RL method, Lower Expectile Q-learning (LEQ), which provides a low-bias model-based value estimation via lower expectile regression of λ-returns. Our empirical results show that LEQ significantly outperforms previous model-based offline RL methods on long-horizon tasks, such as the D4RL AntMaze tasks, matching or surpassing the performance of model-free approaches and sequence modeling approaches. Furthermore, LEQ matches the performance of state-of-the-art model-based and model-free methods in dense-reward environments across both state-based tasks (NeoRL and D4RL) and pixel-based tasks (V-D4RL), showing that LEQ works robustly across diverse domains. Our ablation studies demonstrate that lower expectile regression, λ-returns, and critic training on offline data are all crucial for LEQ.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 47 references that are in the map. First posted on arXiv 30 Jun 2024.

Builds on

39 of 47 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information31 Oct
  2. 2022Efficient Offline Policy Optimization with a Learned Model12 Oct
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  4. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
3 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2023Mastering Diverse Domains through World Modelscited by 1,193
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
12 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  4. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  9. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  10. 2020Model-Based Offline Planningcited by 126
  11. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  12. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  2. 2026Chunk-Guided Q-Learningcited by 3
  3. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learningcited by 2
  4. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  5. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  7. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 1
  8. 2025Controllable Flow Matching for Online Reinforcement Learningcited by 0
  9. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rolloutscited by 0
  10. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  11. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  12. 2026Generative OOD-regularized Model-based Policy Optimizationcited by 0
5 more
  1. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0
  2. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  3. 2026Shortcut Trajectory Planning for Efficient Offline Reinforcement Learningcited by 0
  4. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learningcited by 0
  5. 2026Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  4. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
  5. 2024Strategically Conservative Q-Learning
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  7. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  8. 2025A Clean Slate for Offline Reinforcement Learning