# Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics

arXiv: https://arxiv.org/abs/2402.11317 · HTML: https://mlsanity.com/paper/2402.11317/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 24 of 24)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Prompting Decision Transformer for Few-Shot Policy Generalization](https://mlsanity.com/paper/2206.13499/) (2022)
- [Offline RL Policies Should be Trained to be Adaptive](https://mlsanity.com/paper/2207.02200/) (2022)
- [Policy Consolidation for Continual Reinforcement Learning](https://mlsanity.com/paper/1902.00255/) (2019)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [Policy Regularization with Dataset Constraint for Offline Reinforcement Learning](https://mlsanity.com/paper/2306.06569/) (2023)
- [ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning](https://mlsanity.com/paper/2309.05915/) (2023)
- [Context Shift Reduction for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2311.03695/) (2023)
- [Model-Based Offline Meta-Reinforcement Learning with Regularization](https://mlsanity.com/paper/2202.02929/) (2022)
- [User-Interactive Offline Reinforcement Learning](https://mlsanity.com/paper/2205.10629/) (2022)
- [Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations](https://mlsanity.com/paper/2312.15909/) (2023)

## Built on by (most cited first, 2 of 2)

- [Wavelet Predictive Representations for Non-Stationary Reinforcement Learning](https://mlsanity.com/paper/2510.04507/) (2025)
- [Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2603.02935/) (2026)

## Abstract (arXiv)

Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.