# Transferring Knowledge across Learning Processes

arXiv: https://arxiv.org/abs/1812.01054 · HTML: https://mlsanity.com/paper/1812.01054/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2018.

## Built on by (most cited first, 22 of 22)

- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Transfer Learning with Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2111.11223/) (2021)
- [GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs](https://mlsanity.com/paper/2303.14878/) (2023)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification](https://mlsanity.com/paper/2208.08741/) (2022)
- [Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh](https://mlsanity.com/paper/2211.01604/) (2022)
- [Learning Large-scale Neural Fields via Context Pruned Meta-Learning](https://mlsanity.com/paper/2302.00617/) (2023)
- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Large-Scale Meta-Learning with Continual Trajectory Shifting](https://mlsanity.com/paper/2102.07215/) (2021)
- [Enabling Automatic Differentiation with Mollified Graph Neural Operators](https://mlsanity.com/paper/2504.08277/) (2025)
- [Adaptive Physics-informed Neural Networks: A Survey](https://mlsanity.com/paper/2503.18181/) (2025)
- [Multilingual Speech Recognition using Knowledge Transfer across Learning Processes](https://mlsanity.com/paper/2110.07909/) (2021)
- [Scalable Meta-Learning with Gaussian Processes](https://mlsanity.com/paper/2312.00742/) (2023)
- [Online Hyperparameter Meta-Learning with Hypergradient Distillation](https://mlsanity.com/paper/2110.02508/) (2021)
- [Optimistic Meta-Gradients](https://mlsanity.com/paper/2301.03236/) (2023)
- [Principal component analysis for Gaussian process posteriors](https://mlsanity.com/paper/2107.07115/) (2021)
- [Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution](https://mlsanity.com/paper/2207.08356/) (2022)
- [Gradient-Based Meta-Learning Using Uncertainty to Weigh Loss for Few-Shot Learning](https://mlsanity.com/paper/2208.08135/) (2022)
- [Meta-Learning Neural Procedural Biases](https://mlsanity.com/paper/2406.07983/) (2024)
- [Trajectory-Based Meta-Learning for Out-Of-Vocabulary Word Embedding Learning](https://mlsanity.com/paper/2102.12266/) (2021)
- [Graceful Degradation and Related Fields](https://mlsanity.com/paper/2106.11119/) (2021)
- [Leaping Through Time with Gradient-based Adaptation for Recommendation](https://mlsanity.com/paper/2112.05914/) (2021)

## Abstract (arXiv)

In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at a higher level of abstraction is needed. We propose Leap, a framework that achieves this by transferring knowledge across learning processes. We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path. Our framework leverages only information obtained during training and can be computed on the fly at negligible cost. We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks. Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.