# On Task-Relevant Loss Functions in Meta-Reinforcement Learning and Online LQR

arXiv: https://arxiv.org/abs/2312.05465 · HTML: https://mlsanity.com/paper/2312.05465/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [A Tutorial on Meta-Reinforcement Learning](https://mlsanity.com/paper/2301.08028/) (2023)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://mlsanity.com/paper/2008.02790/) (2020)
- [Meta Reinforcement Learning with Latent Variable Gaussian Processes](https://mlsanity.com/paper/1803.07551/) (2018)
- [The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint](https://mlsanity.com/paper/1812.03565/) (2018)
- [Value Gradient weighted Model-Based Reinforcement Learning](https://mlsanity.com/paper/2204.01464/) (2022)
- [Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads](https://mlsanity.com/paper/2004.11345/) (2020)
- [Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning](https://mlsanity.com/paper/2009.13891/) (2020)
- [Policy-Aware Model Learning for Policy Gradient Methods](https://mlsanity.com/paper/2003.00030/) (2020)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models](https://mlsanity.com/paper/2102.08291/) (2021)
- [Thompson Sampling for Linear-Quadratic Control Problems](https://mlsanity.com/paper/1703.08972/) (2017)
- [Meta-Learning surrogate models for sequential decision making](https://mlsanity.com/paper/1903.11907/) (2019)
- [Meta-Reinforcement Learning in Broad and Non-Parametric Environments](https://mlsanity.com/paper/2108.03718/) (2021)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)

## Built on by (most cited first, 1 of 1)

- [Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.02332/) (2025)

## Abstract (arXiv)

Designing a competent meta-reinforcement learning (meta-RL) algorithm in terms of data usage remains a central challenge to be tackled for its successful real-world applications. In this paper, we propose a sample-efficient meta-RL algorithm that learns a model of the system or environment at hand in a task-directed manner. As opposed to the standard model-based approaches to meta-RL, our method exploits the value information in order to rapidly capture the decision-critical part of the environment. The key component of our method is the loss function for learning the task inference module and the system model that systematically couples the model discrepancy and the value estimate, thereby facilitating the learning of the policy and the task inference module with a significantly smaller amount of data compared to the existing meta-RL algorithms. The idea is also extended to a non-meta-RL setting, namely an online linear quadratic regulator (LQR) problem, where our method can be simplified to reveal the essence of the strategy. The proposed method is evaluated in high-dimensional robotic control and online LQR problems, empirically verifying its effectiveness in extracting information indispensable for solving the tasks from observations in a sample efficient manner.