mlsanityData to 14 Oct 2026

Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning

arXiv 2405.12001 ↗First posted 20 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Offline meta reinforcement learning (OMRL) has emerged as a promising approach for interaction avoidance and strong generalization performance by leveraging pre-collected data and meta-learning techniques. Previous context-based approaches predominantly rely on the intuition that alternating optimization between the context encoder and the policy can lead to performance improvements, as long as the context encoder follows the principle of maximizing the mutual information between the task variable M and its latent representation Z (I(Z;M)) while the policy adopts the standard offline reinforcement learning (RL) algorithms conditioning on the learned task representation.Despite promising results, the theoretical justification of performance improvements for such intuition remains underexplored.Inspired by the return discrepancy scheme in the model-based RL field, we find that the previous optimization framework can be linked with the general RL objective of maximizing the expected return, thereby explaining performance improvements. Furthermore, after scrutinizing this optimization framework, we observe that the condition for monotonic performance improvements does not consider the variation of the task representation. When these variations are considered, the previously established condition may no longer be sufficient to ensure monotonicity, thereby impairing the optimization process.We name this issue task representation shift and theoretically prove that the monotonic performance improvements can be guaranteed with appropriate context encoder updates.Our work opens up a new avenue for OMRL, leading to a better understanding between the task representation and performance improvements.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

39 of 46 references.

References · 39

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
27 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  3. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  6. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  7. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  8. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  9. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  10. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  11. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  12. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  13. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  14. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  15. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  16. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 35
  17. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 31
  18. 2019Efficient Model-free Reinforcement Learning in Metric Spacescited by 18
  19. 2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18
  20. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 18
  21. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 17
  22. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  23. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  24. 2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
  25. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 9
  26. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  27. 2023Safe Reinforcement Learning with Dead-Ends Avoidance and Recoverycited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning
  2. 2021Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
  3. 2020FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization
  4. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
  5. 2023Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations
  6. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation
  7. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  8. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision