Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents
Not on the map. Why?
Abstract (arXiv)
Fast adaptation to new tasks is extremely important for embodied agents in the real world. Meta-reinforcement learning (meta-RL) has emerged as an effective method to enable fast adaptation in unknown environments. Compared to on-policy meta-RL algorithms, off-policy algorithms rely heavily on efficient data sampling strategies to extract and represent the historical trajectories. However, little is known about how different data sampling methods impact the ability of meta-RL agents to represent unknown environments. Here, we investigate the impact of data sampling strategies on the exploration and adaptability of meta-RL agents. Specifically, we conducted experiments with two types of off-policy meta-RL algorithms based on Thompson sampling and Bayes-optimality theories in continuous control tasks within the MuJoCo environment and sparse reward navigation tasks. Our analysis revealed the long-memory and short-memory sequence sampling strategies affect the representation and adaptive capabilities of meta-RL agents. We found that the algorithm based on Bayes-optimality theory exhibited more robust and better adaptability than the algorithm based on Thompson sampling, highlighting the importance of appropriate data sampling strategies for the agent's representation of an unknown environment, especially in the case of sparse rewards.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 18 Jun 2024.
Builds on
22 of 35 references.
References · 22
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2017A Survey on Multi-Task Learningcited by 838
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2018Meta-Learning with Latent Embedding Optimizationcited by 476
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2016Learning to reinforcement learncited by 369
- 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
- 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
- 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
10 more
- 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
- 2023A Tutorial on Meta-Reinforcement Learningcited by 102
- 2019Meta reinforcement learning as task inferencecited by 75
- 2019Meta-learning of Sequential Strategiescited by 68
- 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
- 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
- 2018Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Explorationcited by 23
- 2021Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixturecited by 12
- 2023Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learningcited by 10
- 2021Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learningcited by 8
Built on by
Citing papers per year: 2024: 1.
1 paper.