mlsanityData to 14 Oct 2026

Context Shift Reduction for Offline Meta-Reinforcement Learning

arXiv 2311.03695 ↗First posted 7 Nov 2023cs.LG cs.AIcited by 18

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Offline meta-reinforcement learning (OMRL) utilizes pre-collected offline datasets to enhance the agent's generalization ability on unseen tasks. However, the context shift problem arises due to the distribution discrepancy between the contexts used for training (from the behavior policy) and testing (from the exploration policy). The context shift problem leads to incorrect task inference and further deteriorates the generalization ability of the meta-policy. Existing OMRL methods either overlook this problem or attempt to mitigate it with additional information. In this paper, we propose a novel approach called Context Shift Reduction for OMRL (CSRO) to address the context shift problem with only offline datasets. The key insight of CSRO is to minimize the influence of policy in context during both the meta-training and meta-test phases. During meta-training, we design a max-min mutual information representation learning mechanism to diminish the impact of the behavior policy on task representation. In the meta-test phase, we introduce the non-prior context collection strategy to reduce the effect of the exploration policy. Experimental results demonstrate that CSRO significantly reduces the context shift and improves the generalization ability, surpassing previous methods across various challenging domains.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

29 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptation31 May
  2. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning21 Jun
  3. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization7 Feb
  4. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision8 Jul

Other references · 25

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2016Deep Variational Information Bottleneckcited by 930
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
13 more
  1. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  2. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  3. 2018Meta-Gradient Reinforcement Learningcited by 159
  4. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
  5. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
  6. 2020Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learningcited by 29
  7. 2020MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Explorationcited by 25
  8. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 24
  9. 2023Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learningcited by 9
  10. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  11. 2021Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environmentcited by 7
  12. 2022Object-Category Aware Reinforcement Learningcited by 6
  13. 2023Online Prototype Alignment for Few-shot Policy Transfercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 17
  2. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 17
  3. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  4. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 8
  5. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  6. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  7. 2025Scalable In-Context Q-Learningcited by 5
  8. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  9. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
  10. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agentscited by 1
  11. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
  12. 2024Hierarchical Multi-agent Meta-Reinforcement Learning for Cross-channel Biddingcited by 0
6 more
  1. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learningcited by 0
  2. 2025Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimizationcited by 0
  3. 2025Maximum Total Correlation Reinforcement Learningcited by 0
  4. 2025Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalizationcited by 0
  5. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learningcited by 0
  6. 2026Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
  2. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
  3. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations
  4. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation
  5. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
  6. 2021Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
  7. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  8. 2026Offline Meta-Reinforcement Learning with Flow-Based Task Inference and Adaptive Correction of Feature Overgeneralization