mlsanityData to 14 Oct 2026

Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning

arXiv 2102.10774 ↗First posted 22 Feb 2021cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Meta-learning for offline reinforcement learning (OMRL) is an understudied problem with tremendous potential impact by enabling RL algorithms in many real-world applications. A popular solution to the problem is to infer task identity as augmented state using a context-based encoder, for which efficient learning of robust task representations remains an open challenge. In this work, we provably improve upon one of the SOTA OMRL algorithms, FOCAL, by incorporating intra-task attention mechanism and inter-task contrastive learning objectives, to robustify task representation learning against sparse reward and distribution shift. Theoretical analysis and experiments are presented to demonstrate the superior performance and robustness of our end-to-end and model-free framework compared to prior algorithms across multiple meta-RL benchmarks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 58 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

41 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization2 Oct
  2. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2017Graph Attention Networkscited by 8,923
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2017Squeeze-and-Excitation Networkscited by 4,789
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
27 more
  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  7. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  8. 2016Learning to reinforcement learncited by 377
  9. 2014Recurrent Models of Visual Attentioncited by 349
  10. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  11. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  12. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  13. 2015Contextual Markov Decision Processescited by 220
  14. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
  15. 2019Adaptive Attention Span in Transformerscited by 149
  16. 2019Finding Task-Relevant Features for Few-Shot Learning by Category Traversalcited by 101
  17. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  18. 2019Meta-Learning without Memorizationcited by 92
  19. 2019Observational Overfitting in Reinforcement Learningcited by 91
  20. 2017Deep Visual Attention Predictioncited by 70
  21. 2019Meta-Q-Learningcited by 62
  22. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 60
  23. 2020Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learningcited by 29
  24. 2020Adaptive Transformers in RLcited by 13
  25. 2020Fast Adaptation via Policy-Dynamics Value Functionscited by 13
  26. 2020Offline Meta Learning of Explorationcited by 13
  27. 2019An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environmentscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.

2 papers.

  1. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representationcited by 3
  2. 2022Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning
  2. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning
  3. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
  4. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations
  5. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
  6. 2023Context Shift Reduction for Offline Meta-Reinforcement Learning
  7. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
  8. 2022Contextual Transformer for Offline Meta Reinforcement Learning