mlsanityData to 14 Oct 2026

Offline Reinforcement Learning from Datasets with Structured Non-Stationarity

arXiv 2405.14114 ↗First posted 23 May 2024cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. Offline RL aims to solve this issue by using transitions collected by a different behavior policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation. We analyze our proposed method and show that it performs well in simple continuous control tasks and challenging, high-dimensional locomotion tasks. We show that our method often achieves the oracle performance and performs better than baselines.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 43 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

31 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  3. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  4. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  5. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2016Layer Normalizationcited by 5,669
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  12. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
10 more
  1. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  2. 2019Contrastive Learning of Structured World Modelscited by 209
  3. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  4. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  5. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  6. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  7. 2023Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learningcited by 10
  8. 2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
  9. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  10. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  2. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  3. 2024Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTCcited by 0
  4. 2025Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measurescited by 0
  5. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning as Anti-Exploration
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  5. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2021A Dataset Perspective on Offline Reinforcement Learning