# FOCAL: Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization

arXiv: https://arxiv.org/abs/2010.01112 · HTML: https://mlsanity.com/paper/2010.01112/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 2 Oct 2020.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 25 of 25)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning](https://mlsanity.com/paper/2002.08396/) (2020)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Finding Task-Relevant Features for Few-Shot Learning by Category Traversal](https://mlsanity.com/paper/1905.11116/) (2019)
- [Meta-Learning without Memorization](https://mlsanity.com/paper/1912.03820/) (2019)
- [Evolved Policy Gradients](https://mlsanity.com/paper/1802.04821/) (2018)
- [Meta-Q-Learning](https://mlsanity.com/paper/1910.00125/) (2019)
- [A Simple Neural Attentive Meta-Learner](https://mlsanity.com/paper/1707.03141/) (2017)
- [Offline Meta Learning of Exploration](https://mlsanity.com/paper/2008.02598/) (2020)

## Built on by (most cited first, 7 of 7)

- [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing](https://mlsanity.com/paper/2206.02829/) (2022)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [Model-Based Offline Meta-Reinforcement Learning with Regularization](https://mlsanity.com/paper/2202.02929/) (2022)
- [MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance](https://mlsanity.com/paper/2112.03575/) (2021)
- [Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning](https://mlsanity.com/paper/2102.10774/) (2021)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2606.00780/) (2026)

## Abstract (arXiv)

We study the offline meta-reinforcement learning (OMRL) problem, a paradigm which enables reinforcement learning (RL) algorithms to quickly adapt to unseen tasks without any interactions with the environments, making RL truly practical in many real-world applications. This problem is still not fully understood, for which two major challenges need to be addressed. First, offline RL usually suffers from bootstrapping errors of out-of-distribution state-actions which leads to divergence of value functions. Second, meta-RL requires efficient and robust task inference learned jointly with control policy. In this work, we enforce behavior regularization on learned policy as a general approach to offline RL, combined with a deterministic context encoder for efficient task inference. We propose a novel negative-power distance metric on bounded context embedding space, whose gradients propagation is detached from the Bellman backup. We provide analysis and insight showing that some simple design choices can yield substantial improvements over recent approaches involving meta-RL and distance metric learning. To the best of our knowledge, our method is the first model-free and end-to-end OMRL algorithm, which is computationally efficient and demonstrated to outperform prior algorithms on several meta-RL benchmarks.