mlsanityData to 7 Oct 2026

Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning

arXiv 2402.02429 ↗First posted 4 Feb 2024cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

As a marriage between offline RL and meta-RL, the advent of offline meta-reinforcement learning (OMRL) has shown great promise in enabling RL agents to multi-task and quickly adapt while acquiring knowledge safely. Among which, context-based OMRL (COMRL) as a popular paradigm, aims to learn a universal policy conditioned on effective task representations. In this work, by examining several key milestones in the field of COMRL, we propose to integrate these seemingly independent methodologies into a unified framework. Most importantly, we show that the pre-existing COMRL algorithms are essentially optimizing the same mutual information objective between the task variable M and its latent representation Z by implementing various approximate bounds. Such theoretical insight offers ample design freedom for novel algorithms. As demonstrations, we propose a supervised and a self-supervised implementation of I(Z; M), and empirically show that the corresponding optimization algorithms exhibit remarkable generalization across a broad spectrum of RL benchmarks, context shift scenarios, data qualities and deep learning architectures. This work lays the information theoretic foundation for COMRL methods, leading to a better understanding of task representation learning in the context of reinforcement learning. Given its generality, we envision our framework as a promising offline pre-training paradigm of foundation models for decision making.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 69 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

50 of 69 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations26 Dec
  2. 2023Context Shift Reduction for Offline Meta-Reinforcement Learning7 Nov
  3. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining12 Oct
  4. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  5. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning21 Jun
  6. 2022Multi-Game Decision Transformers30 May
  7. 2021Value Penalized Q-Learning for Recommender Systems15 Oct
  8. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision8 Jul
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun
  11. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 39

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017Improved Training of Wasserstein GANscited by 2,223
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2018World Modelscited by 1,444
27 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2016Deep Variational Information Bottleneckcited by 915
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2022A Generalist Agentcited by 802
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  7. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  10. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  12. 2016Learning to reinforcement learncited by 369
  13. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  14. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  15. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  16. 2015Contextual Markov Decision Processescited by 213
  17. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  18. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  19. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  20. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  21. 2019Meta-Q-Learningcited by 62
  22. 2021Offline Meta-Reinforcement Learning for Industrial Insertioncited by 37
  23. 2017A Simple Neural Attentive Meta-Learnercited by 22
  24. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
  25. 2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
  26. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  27. 2022Designing Biological Sequences via Meta-Reinforcement Learning and Bayesian Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  2. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  3. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
  4. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  5. 2025Scalable In-Context Q-Learningcited by 5
  6. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  7. 2025Is Meta-Learning Out? Rethinking Unsupervised Few-Shot Classification with Limited Entropycited by 4
  8. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representationcited by 2
  9. 2025Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPscited by 2
  10. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  11. 2025Offline Diffusion Policy for Multi-User Delay-Constrained Schedulingcited by 1
  12. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
4 more
  1. 2025Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspectivecited by 0
  2. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0
  3. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learningcited by 0
  4. 2026MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
  2. 2021Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
  3. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
  4. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
  5. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  6. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
  7. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  8. 2026Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning