Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics
Not on the map. Why?
Abstract (arXiv)
Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 17 Feb 2024.
Builds on
24 of 41 references.
References · 24
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2015Deep Learning and the Information Bottleneck Principlecited by 793
- 2019Model-Based Reinforcement Learning for Ataricited by 495
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
12 more
- 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
- 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
- 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
- 2019Policy Consolidation for Continual Reinforcement Learningcited by 37
- 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
- 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
- 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
- 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 17
- 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
- 2022User-Interactive Offline Reinforcement Learningcited by 11
- 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitationscited by 9
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.