mlsanityData to 7 Oct 2026

Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning

arXiv 2206.10442 ↗First posted 21 Jun 2022cs.LGcited by 33

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

We study offline meta-reinforcement learning, a practical reinforcement learning paradigm that learns from offline data to adapt to new tasks. The distribution of offline data is determined jointly by the behavior policy and the task. Existing offline meta-reinforcement learning algorithms cannot distinguish these factors, making task representations unstable to the change of behavior policies. To address this problem, we propose a contrastive learning framework for task representations that are robust to the distribution mismatch of behavior policies in training and test. We design a bi-level encoder structure, use mutual information maximization to formalize task representation learning, derive a contrastive learning objective, and introduce several approaches to approximate the true distribution of negative pairs. Experiments on a variety of offline meta-reinforcement learning benchmarks demonstrate the advantages of our method over prior methods, especially on the generalization to out-of-distribution behavior policies. The code is available at https://github.com/PKU-AI-Edge/CORRO.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 52 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

39 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization7 Feb
  2. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision8 Jul
  3. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2019Contrastive Multiview Codingcited by 1,297
24 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2019Model-Based Reinforcement Learning for Ataricited by 495
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  10. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  11. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  12. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  13. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
  14. 2019Unsupervised State Representation Learning in Ataricited by 144
  15. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  16. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  17. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  18. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  19. 2018Evolved Policy Gradientscited by 86
  20. 2019Meta-Q-Learningcited by 62
  21. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  22. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  23. 2020P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understandingcited by 41
  24. 2019Concurrent Meta Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 13; 2025: 8; 2026: 3.

33 papers.

Sort:

By area:

  1. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  2. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 65
  3. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  4. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  5. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 17
  6. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  7. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  8. 2023A Survey of Demonstration Learningcited by 12
  9. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
  10. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptationcited by 10
  11. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitationscited by 9
  12. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
21 more
  1. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  2. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  3. 2025Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learningcited by 5
  4. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  5. 2025Scalable In-Context Q-Learningcited by 5
  6. 2023On Context Distribution Shift in Task Representation Learning for Offline Meta RLcited by 4
  7. 2023Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skillscited by 4
  8. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  9. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  10. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
  11. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentationcited by 2
  12. 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learningcited by 1
  13. 2024Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agentscited by 1
  14. 2026DADP: Domain Adaptive Diffusion Policycited by 1
  15. 2023Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Makingcited by 0
  16. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
  17. 2024CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learningcited by 0
  18. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
  19. 2025Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learningcited by 0
  20. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learningcited by 0
  21. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
  2. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
  3. 2026Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning
  4. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
  5. 2024Offline Multi-task Transfer RL with Representational Penalization
  6. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  7. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation
  8. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning