On Task-Relevant Loss Functions in Meta-Reinforcement Learning and Online LQR
Not on the map. Why?
Abstract (arXiv)
Designing a competent meta-reinforcement learning (meta-RL) algorithm in terms of data usage remains a central challenge to be tackled for its successful real-world applications. In this paper, we propose a sample-efficient meta-RL algorithm that learns a model of the system or environment at hand in a task-directed manner. As opposed to the standard model-based approaches to meta-RL, our method exploits the value information in order to rapidly capture the decision-critical part of the environment. The key component of our method is the loss function for learning the task inference module and the system model that systematically couples the model discrepancy and the value estimate, thereby facilitating the learning of the policy and the task inference module with a significantly smaller amount of data compared to the existing meta-RL algorithms. The idea is also extended to a non-meta-RL setting, namely an online linear quadratic regulator (LQR) problem, where our method can be simplified to reveal the essence of the strategy. The proposed method is evaluated in high-dimensional robotic control and online LQR problems, empirically verifying its effectiveness in extracting information indispensable for solving the tasks from observations in a sample efficient manner.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 9 Dec 2023.
Builds on
33 of 45 references.
References · 33
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
- 2016Learning to reinforcement learncited by 377
- 2020dm_control: Software and Tasks for Continuous Controlcited by 375
- 2019A Theory of Regularized Markov Decision Processescited by 286
- 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
- 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
- 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
21 more
- 2020Naive Exploration is Optimal for Online LQRcited by 116
- 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
- 2023A Tutorial on Meta-Reinforcement Learningcited by 108
- 2020Objective Mismatch in Model-based Reinforcement Learningcited by 96
- 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
- 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
- 2018Meta Reinforcement Learning with Latent Variable Gaussian Processescited by 44
- 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 40
- 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 32
- 2020Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloadscited by 30
- 2020Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learningcited by 29
- 2020Policy-Aware Model Learning for Policy Gradient Methodscited by 23
- 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
- 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 17
- 2017Thompson Sampling for Linear-Quadratic Control Problemscited by 14
- 2019Meta-Learning surrogate models for sequential decision makingcited by 14
- 2021Meta-Reinforcement Learning in Broad and Non-Parametric Environmentscited by 10
- 2023Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learningcited by 10
- 2016Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Modelscited by 5
- 2021Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environmentscited by 5
- 2018Efficient transfer learning and online adaptation with latent variable models for continuous controlcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.