mlsanityData to 14 Oct 2026

Forecasting in Offline Reinforcement Learning for Non-stationary Environments

arXiv 2512.01987 ↗First posted 1 Dec 2025cs.LG cs.AI cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 84 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

58 of 84 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  2. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity23 May
  3. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximation23 Oct
  4. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  5. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  6. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions29 Nov
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
3 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 43

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  11. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
31 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  4. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  5. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  6. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 395
  7. 2024One Step Diffusion via Shortcut Modelscited by 391
  8. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  11. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  12. 2021Monash Time Series Forecasting Archivecited by 267
  13. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  14. 2025Flow Q-Learningcited by 184
  15. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 149
  16. 2019High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processescited by 148
  17. 2019Online Meta-Learningcited by 137
  18. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  19. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  20. 2020Invariant Causal Prediction for Block MDPscited by 105
  21. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  22. 2019GluonTS: Probabilistic Time Series Models in Pythoncited by 46
  23. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 42
  24. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  25. 2024Reward Guided Latent Consistency Distillationcited by 35
  26. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  27. 2022AnyMorph: Learning Transferable Polices By Inferring Agent Morphologycited by 27
  28. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 26
  29. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  30. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  31. 2025Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  2. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
  6. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  7. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  8. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization