# On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning

arXiv: https://arxiv.org/abs/2206.03271 · HTML: https://mlsanity.com/paper/2206.03271/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 80 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 61)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)

## Built on by (most cited first, 19 of 19)

- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Learning to Modulate pre-trained Models in RL](https://mlsanity.com/paper/2306.14884/) (2023)
- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Accelerating Exploration with Unlabeled Prior Data](https://mlsanity.com/paper/2311.05067/) (2023)
- [Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.02429/) (2024)
- [AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers](https://mlsanity.com/paper/2411.11188/) (2024)
- [SIRL: Similarity-based Implicit Representation Learning](https://mlsanity.com/paper/2301.00810/) (2023)
- [MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning](https://mlsanity.com/paper/2403.09859/) (2024)
- [Meta-Learning via Classifier(-free) Diffusion Guidance](https://mlsanity.com/paper/2210.08942/) (2022)
- [PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining](https://mlsanity.com/paper/2303.08789/) (2023)
- [Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach](https://mlsanity.com/paper/2206.10716/) (2022)
- [Test-Time Regret Minimization in Meta Reinforcement Learning](https://mlsanity.com/paper/2406.02282/) (2024)
- [Redundancy-aware Action Spaces for Robot Learning](https://mlsanity.com/paper/2406.04144/) (2024)
- [RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning](https://mlsanity.com/paper/2401.14858/) (2024)
- [Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation](https://mlsanity.com/paper/2306.13554/) (2023)
- [ProMi: An Efficient Prototype-Mixture Baseline for Few-Shot Segmentation with Bounding-Box Annotations](https://mlsanity.com/paper/2505.12547/) (2025)
- [Few-shot Scooping Under Domain Shift via Simulated Maximal Deployment Gaps](https://mlsanity.com/paper/2408.02949/) (2024)
- [In-Context Exploiter for Extensive-Form Games](https://mlsanity.com/paper/2408.05575/) (2024)
- [Task Specialization Fine-Tuning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2608.17180/) (2026)

## Abstract (arXiv)

Intelligent agents should have the ability to leverage knowledge from previously learned tasks in order to learn new ones quickly and efficiently. Meta-learning approaches have emerged as a popular solution to achieve this. However, meta-reinforcement learning (meta-RL) algorithms have thus far been restricted to simple environments with narrow task distributions. Moreover, the paradigm of pretraining followed by fine-tuning to adapt to new tasks has emerged as a simple yet effective solution in supervised and self-supervised learning. This calls into question the benefits of meta-learning approaches also in reinforcement learning, which typically come at the cost of high complexity. We hence investigate meta-RL approaches in a variety of vision-based benchmarks, including Procgen, RLBench, and Atari, where evaluations are made on completely novel tasks. Our findings show that when meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pretraining with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, as multi-task pretraining tends to be simpler and computationally cheaper than meta-RL. From these findings, we advocate for evaluating future meta-RL methods on more challenging tasks and including multi-task pretraining with fine-tuning as a simple, yet strong baseline.