Data to 7 Oct 2026

Offline Meta-Reinforcement Learning with Advantage Weighting

arXiv 2008.06043 ↗First posted 13 Aug 2020cs.LG cs.AI stat.MLcited by 92

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre-training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model to a new task with relatively little data. That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offline, algorithms for offline meta-RL can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. Targeting the offline meta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW), an optimization-based meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offline variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 51 references that are in the map. First posted on arXiv 13 Aug 2020.

Builds on

39 of 51 references.

References · 39

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2016Matching Networks for One Shot Learningcited by 2,246
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
2015End-to-End Training of Deep Visuomotor Policiescited by 847
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750

27 more

2019Behavior Regularized Offline Reinforcement Learningcited by 654
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2019Meta-Learning with Implicit Gradientscited by 523
2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
2018Meta-Learning with Latent Embedding Optimizationcited by 476
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
2016Learning to reinforcement learncited by 369
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
2019Generalized Inner Loop Meta-Learningcited by 92
2018ProMP: Proximal Meta-Policy Searchcited by 91
2018Evolved Policy Gradientscited by 86
2019Meta reinforcement learning as task inferencecited by 75
2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
2019Improving Generalization in Meta Reinforcement Learning using Learned Objectivescited by 68
2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
2019Meta-Q-Learningcited by 62
2019Meta-Learning via Learned Losscited by 60
2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
2019Guided Meta-Policy Searchcited by 26
2017A Simple Neural Attentive Meta-Learnercited by 22
2020Offline Meta Learning of Explorationcited by 12
2019NoRML: No-Reward Meta Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 19; 2023: 19; 2024: 27; 2025: 14; 2026: 5.

92 papers.

Sort:

By area:

2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
2023Elastic Decision Transformercited by 66
2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 65
2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39

38 more

2021Offline Meta-Reinforcement Learning for Industrial Insertioncited by 37
2022Skill-based Meta-Reinforcement Learningcited by 36
2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 25
2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modelingcited by 18
2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
2021CoMPS: Continual Meta Policy Searchcited by 15
2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
2022Distributionally Adaptive Meta Reinforcement Learningcited by 11
2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptationcited by 10
2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
2022Learning Robust Real-Time Cultural Transmission without Human Datacited by 9
2023Hacking Task Confounder in Meta-Learningcited by 8
2021MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerancecited by 7
2022Contextual Transformer for Offline Meta Reinforcement Learningcited by 7
2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
2021On the Practical Consistency of Meta-Reinforcement Learning Algorithmscited by 6
2022Simple Emergent Action Representations from Multi-Task Policy Trainingcited by 6
2024Solving Continual Offline Reinforcement Learning with Decision Transformercited by 6
2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way