mlsanityData to 7 Oct 2026

Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learning

arXiv 2406.04815 ↗First posted 7 Jun 2024cs.LG cs.AI cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Meta-Reinforcement Learning (Meta-RL) agents can struggle to operate across tasks with varying environmental features that require different optimal skills (i.e., different modes of behaviour). Using context encoders based on contrastive learning to enhance the generalisability of Meta-RL agents is now widely studied but faces challenges such as the requirement for a large sample size, also referred to as the log-K curse. To improve RL generalisation to different tasks, we first introduce Skill-aware Mutual Information (SaMI), an optimisation objective that aids in distinguishing context embeddings according to skills, thereby equipping RL agents with the ability to identify and execute different skills across tasks. We then propose Skill-aware Noise Contrastive Estimation (SaNCE), a K-sample estimator used to optimise the SaMI objective. We provide a framework for equipping an RL agent with SaNCE in practice and conduct experimental validation on modified MuJoCo and Panda-gym benchmarks. We empirically find that RL agents that learn by maximising SaMI achieve substantially improved zero-shot generalisation to unseen tasks. Additionally, the context encoder trained with SaNCE demonstrates greater robustness to a reduction in the number of available samples, thus possessing the potential to overcome the log-K curse.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

39 of 57 references.

References · 39

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  12. 2020Contrastive Learning with Hard Negative Samplescited by 619
27 more
  1. 2019On Variational Bounds of Mutual Informationcited by 550
  2. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  3. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  4. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  5. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  6. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  7. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  8. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  9. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  10. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  11. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  12. 2022Skill-based Model-based Reinforcement Learningcited by 50
  13. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  14. 2019Environment Probing Interaction Policiescited by 39
  15. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 34
  16. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 31
  17. 2020Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learningcited by 29
  18. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  19. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  20. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  21. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  22. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  23. 2021Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learningcited by 8
  24. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  25. 2018Learning Invariances for Policy Generalizationcited by 7
  26. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  27. 2022PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representationscited by 6

Built on by

Not cited yet.