mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Universal Horizon Models

arXiv 2605.15603 ↗First posted 15 May 2026cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate this issue through direct prediction over a discounted infinite-horizon future, they remain challenged in accurately modeling distant future states. To this end, we introduce universal horizon models (UHM), a generalization of GHM that directly predicts future states under arbitrary horizons. Leveraging this flexibility, we propose a scalable value learning method that employs a winsorized horizon distribution to stabilize training by capping excessively large horizons. Experimental results on 100 challenging OGBench tasks demonstrate that the proposed method outperforms competitive baselines, particularly on tasks with highly suboptimal datasets and those requiring long-horizon reasoning. Project page: https://rllab-snu.github.io/projects/UHM/

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 58 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

49 of 58 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Scalable Offline Model-Based RL with Action Chunks8 Dec
  2. 2025Reinforcement Learning with Action Chunking10 Jul
  3. 2025Horizon Reduction Makes RL Scalable4 Jun
  4. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining1 Oct
  5. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  6. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning27 May
  7. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  9. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  10. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  11. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
6 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 31

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  12. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
19 more
  1. 2025Flow Q-Learningcited by 184
  2. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  3. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  4. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  5. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  6. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  7. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  8. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  9. 2024Policy-Guided Diffusioncited by 41
  10. 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
  11. 2025Temporal Difference Flowscited by 24
  12. 2025Decoupled Q-Chunkingcited by 20
  13. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13
  14. 2023Diffused Task-Agnostic Milestone Plannercited by 12
  15. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  16. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  17. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  18. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  19. 2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learningcited by 0
  2. 2026Imagining Goals with Geometric Horizon Models for Policy Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Horizon Adaptive Offline Policy Learning via Value Stitching
  2. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
  3. 2020Offline Reinforcement Learning from Images with Latent Space Models
  4. 2022Offline Reinforcement Learning with Causal Structured World Models
  5. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  6. 2025Horizon Reduction as Information Loss in Offline Reinforcement Learning
  7. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  8. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL