# Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning

arXiv: https://arxiv.org/abs/2206.10442 · HTML: https://mlsanity.com/paper/2206.10442/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 39 of its 52 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [A Review of Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03963/) (2019)
- [Unsupervised State Representation Learning in Atari](https://mlsanity.com/paper/1906.08226/) (2019)
- [Benchmarking Batch Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1910.01708/) (2019)
- [Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments](https://mlsanity.com/paper/1710.03641/) (2017)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)

## Built on by (most cited first, 30 of 35)

- [Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2305.18459/) (2023)
- [MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL](https://mlsanity.com/paper/2305.19923/) (2023)
- [Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings](https://mlsanity.com/paper/2402.17135/) (2024)
- [Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training](https://mlsanity.com/paper/2402.14407/) (2024)
- [Context Shift Reduction for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2311.03695/) (2023)
- [Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.02429/) (2024)
- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning](https://mlsanity.com/paper/2502.19009/) (2025)
- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Offline Meta Reinforcement Learning with In-Distribution Online Adaptation](https://mlsanity.com/paper/2305.19529/) (2023)
- [Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations](https://mlsanity.com/paper/2312.15909/) (2023)
- [Mixture-of-Experts Meets In-Context Reinforcement Learning](https://mlsanity.com/paper/2506.05426/) (2025)
- [Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay](https://mlsanity.com/paper/2404.10662/) (2024)
- [Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer](https://mlsanity.com/paper/2408.01402/) (2024)
- [Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.03752/) (2025)
- [Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision](https://mlsanity.com/paper/2504.15046/) (2025)
- [Scalable In-Context Q-Learning](https://mlsanity.com/paper/2506.01299/) (2025)
- [On Context Distribution Shift in Task Representation Learning for Offline Meta RL](https://mlsanity.com/paper/2304.00354/) (2023)
- [Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills](https://mlsanity.com/paper/2312.06518/) (2023)
- [Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks](https://mlsanity.com/paper/2502.02834/) (2025)
- [CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries](https://mlsanity.com/paper/2506.00388/) (2025)
- [Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics](https://mlsanity.com/paper/2402.11317/) (2024)
- [Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation](https://mlsanity.com/paper/2403.07261/) (2024)
- [Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning](https://mlsanity.com/paper/2402.02772/) (2024)
- [Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents](https://mlsanity.com/paper/2406.12359/) (2024)
- [DADP: Domain Adaptive Diffusion Policy](https://mlsanity.com/paper/2602.04037/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making](https://mlsanity.com/paper/2312.11027/) (2023)
- [Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2405.12001/) (2024)
- [CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learning](https://mlsanity.com/paper/2408.05781/) (2024)

## Abstract (arXiv)

We study offline meta-reinforcement learning, a practical reinforcement learning paradigm that learns from offline data to adapt to new tasks. The distribution of offline data is determined jointly by the behavior policy and the task. Existing offline meta-reinforcement learning algorithms cannot distinguish these factors, making task representations unstable to the change of behavior policies. To address this problem, we propose a contrastive learning framework for task representations that are robust to the distribution mismatch of behavior policies in training and test. We design a bi-level encoder structure, use mutual information maximization to formalize task representation learning, derive a contrastive learning objective, and introduce several approaches to approximate the true distribution of negative pairs. Experiments on a variety of offline meta-reinforcement learning benchmarks demonstrate the advantages of our method over prior methods, especially on the generalization to out-of-distribution behavior policies. The code is available at https://github.com/PKU-AI-Edge/CORRO.