# Meta-Q-Learning

arXiv: https://arxiv.org/abs/1910.00125 · HTML: https://mlsanity.com/paper/1910.00125/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 30 of 62)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models](https://mlsanity.com/paper/2310.05905/) (2023)
- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning](https://mlsanity.com/paper/2206.10442/) (2022)
- [On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning](https://mlsanity.com/paper/2206.03271/) (2022)
- [Collect & Infer -- a fresh look at data-efficient Reinforcement Learning](https://mlsanity.com/paper/2108.10273/) (2021)
- [Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.02429/) (2024)
- [Safe Exploration by Solving Early Terminated MDP](https://mlsanity.com/paper/2107.04200/) (2021)
- [AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers](https://mlsanity.com/paper/2411.11188/) (2024)
- [EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data](https://mlsanity.com/paper/2406.17768/) (2024)
- [Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture](https://mlsanity.com/paper/2105.13524/) (2021)
- [Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges](https://mlsanity.com/paper/2205.14495/) (2022)
- [MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks](https://mlsanity.com/paper/2506.05982/) (2025)
- [Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning](https://mlsanity.com/paper/2209.13220/) (2022)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Efficient Symbolic Policy Learning with Differentiable Symbolic Expression](https://mlsanity.com/paper/2311.02104/) (2023)
- [Linear Representation Meta-Reinforcement Learning for Instant Adaptation](https://mlsanity.com/paper/2101.04750/) (2021)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [ContraBAR: Contrastive Bayes-Adaptive Deep RL](https://mlsanity.com/paper/2306.02418/) (2023)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [AlphaRouter: Quantum Circuit Routing with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2410.05115/) (2024)
- [FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization](https://mlsanity.com/paper/2010.01112/) (2020)
- [Faster Deep Reinforcement Learning with Slower Online Network](https://mlsanity.com/paper/2112.05848/) (2021)

## Abstract (arXiv)

This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.