# Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.02429 · HTML: https://mlsanity.com/paper/2402.02429/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 50 of its 69 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 30 of 50)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)

## Built on by (most cited first, 17 of 17)

- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [Mixture-of-Experts Meets In-Context Reinforcement Learning](https://mlsanity.com/paper/2506.05426/) (2025)
- [Focus On What Matters: Separated Models For Visual-Based RL Generalization](https://mlsanity.com/paper/2410.10834/) (2024)
- [Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision](https://mlsanity.com/paper/2504.15046/) (2025)
- [Scalable In-Context Q-Learning](https://mlsanity.com/paper/2506.01299/) (2025)
- [Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformers](https://mlsanity.com/paper/2504.04395/) (2025)
- [Is Meta-Learning Out? Rethinking Unsupervised Few-Shot Classification with Limited Entropy](https://mlsanity.com/paper/2509.13185/) (2025)
- [CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation](https://mlsanity.com/paper/2502.00983/) (2025)
- [PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning](https://mlsanity.com/paper/2511.13765/) (2025)
- [Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPs](https://mlsanity.com/paper/2510.01620/) (2025)
- [Offline Diffusion Policy for Multi-User Delay-Constrained Scheduling](https://mlsanity.com/paper/2501.12942/) (2025)
- [Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2405.12001/) (2024)
- [Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective](https://mlsanity.com/paper/2506.16288/) (2025)
- [Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space](https://mlsanity.com/paper/2509.25876/) (2025)
- [Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2603.02935/) (2026)
- [MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddings](https://mlsanity.com/paper/2605.17431/) (2026)
- [Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2610.15428/) (2026)

## Abstract (arXiv)

As a marriage between offline RL and meta-RL, the advent of offline meta-reinforcement learning (OMRL) has shown great promise in enabling RL agents to multi-task and quickly adapt while acquiring knowledge safely. Among which, context-based OMRL (COMRL) as a popular paradigm, aims to learn a universal policy conditioned on effective task representations. In this work, by examining several key milestones in the field of COMRL, we propose to integrate these seemingly independent methodologies into a unified framework. Most importantly, we show that the pre-existing COMRL algorithms are essentially optimizing the same mutual information objective between the task variable M and its latent representation Z by implementing various approximate bounds. Such theoretical insight offers ample design freedom for novel algorithms. As demonstrations, we propose a supervised and a self-supervised implementation of I(Z; M), and empirically show that the corresponding optimization algorithms exhibit remarkable generalization across a broad spectrum of RL benchmarks, context shift scenarios, data qualities and deep learning architectures. This work lays the information theoretic foundation for COMRL methods, leading to a better understanding of task representation learning in the context of reinforcement learning. Given its generality, we envision our framework as a promising offline pre-training paradigm of foundation models for decision making.