# Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning

arXiv: https://arxiv.org/abs/2009.13891 · HTML: https://mlsanity.com/paper/2009.13891/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 29 of 29)

- [Contextualize Me -- The Case for Context in Reinforcement Learning](https://mlsanity.com/paper/2202.04500/) (2022)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [Context Shift Reduction for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2311.03695/) (2023)
- [Contrastive Reinforcement Learning of Symbolic Reasoning Domains](https://mlsanity.com/paper/2106.09146/) (2021)
- [Simplifying Polylogarithms with Machine Learning](https://mlsanity.com/paper/2206.04115/) (2022)
- [A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning](https://mlsanity.com/paper/2112.15400/) (2021)
- [Model-based Lifelong Reinforcement Learning with Bayesian Exploration](https://mlsanity.com/paper/2210.11579/) (2022)
- [Distributionally Adaptive Meta Reinforcement Learning](https://mlsanity.com/paper/2210.03104/) (2022)
- [Efficient Symbolic Policy Learning with Differentiable Symbolic Expression](https://mlsanity.com/paper/2311.02104/) (2023)
- [Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations](https://mlsanity.com/paper/2312.15909/) (2023)
- [Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate](https://mlsanity.com/paper/2405.15384/) (2024)
- [Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning](https://mlsanity.com/paper/2103.06386/) (2021)
- [ContraBAR: Contrastive Bayes-Adaptive Deep RL](https://mlsanity.com/paper/2306.02418/) (2023)
- [Meta-Learning Parameterized Skills](https://mlsanity.com/paper/2206.03597/) (2022)
- [Contextual Transformer for Offline Meta Reinforcement Learning](https://mlsanity.com/paper/2211.08016/) (2022)
- [Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2312.10385/) (2023)
- [PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representations](https://mlsanity.com/paper/2204.02877/) (2022)
- [Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching](https://mlsanity.com/paper/2311.01331/) (2023)
- [First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs](https://mlsanity.com/paper/2307.02276/) (2023)
- [Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction](https://mlsanity.com/paper/2103.02225/) (2021)
- [Meta Reinforcement Learning with Successor Feature Based Context](https://mlsanity.com/paper/2207.14723/) (2022)
- [Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning](https://mlsanity.com/paper/2211.12774/) (2022)
- [Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks](https://mlsanity.com/paper/2502.02834/) (2025)
- [Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning](https://mlsanity.com/paper/2102.10774/) (2021)
- [Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learning](https://mlsanity.com/paper/2210.04209/) (2022)
- [On Task-Relevant Loss Functions in Meta-Reinforcement Learning and Online LQR](https://mlsanity.com/paper/2312.05465/) (2023)
- [Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2406.04815/) (2024)
- [Learning Embeddings for Sequential Tasks Using Population of Agents](https://mlsanity.com/paper/2306.03311/) (2023)
- [T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler](https://mlsanity.com/paper/2608.30765/) (2026)

## Abstract (arXiv)

Context, the embedding of previous collected trajectories, is a powerful construct for Meta-Reinforcement Learning (Meta-RL) algorithms. By conditioning on an effective context, Meta-RL policies can easily generalize to new tasks within a few adaptation steps. We argue that improving the quality of context involves answering two questions: 1. How to train a compact and sufficient encoder that can embed the task-specific information contained in prior trajectories? 2. How to collect informative trajectories of which the corresponding context reflects the specification of tasks? To this end, we propose a novel Meta-RL framework called CCM (Contrastive learning augmented Context-based Meta-RL). We first focus on the contrastive nature behind different tasks and leverage it to train a compact and sufficient context encoder. Further, we train a separate exploration policy and theoretically derive a new information-gain-based objective which aims to collect informative trajectories in a few steps. Empirically, we evaluate our approaches on common benchmarks as well as several complex sparse-reward environments. The experimental results show that CCM outperforms state-of-the-art algorithms by addressing previously mentioned problems respectively.