Towards Enabling Meta-Learning from Target Models
Not on the map. Why?
Abstract (arXiv)
Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping mathitsupport sets and mathitquery sets to train and evaluate the solvers respectively due to simplicity (S/Q protocol). Different from S/Q protocol, we can also evaluate a task-specific solver by comparing it to a target model T, which is the optimal model for this task or a model that behaves well enough on this task (S/T protocol). Although being short of research, S/T protocol has unique advantages such as offering more informative supervision, but it is computationally expensive. This paper looks into this special evaluation method and takes a step towards putting it into practice. We find that with a small ratio of tasks armed with target models, classic meta-learning algorithms can be improved a lot without consuming many resources. We empirically verify the effectiveness of S/T protocol in a typical application of meta-learning, mathiti.e., few-shot learning. In detail, after constructing target models by fine-tuning the pre-trained network on those hard tasks, we match the task-specific solvers and target models via knowledge distillation.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 8 Apr 2021.
Builds on
27 of 42 references.
References · 27
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2017Prototypical Networks for Few-shot Learningcited by 3,001
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2014Neural Turing Machinescited by 630
- 2018Meta-Learning with Latent Embedding Optimizationcited by 476
- 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
- 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
- 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
- 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
15 more
- 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
- 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
- 2021Free Lunch for Few-shot Learning: Distribution Calibrationcited by 165
- 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
- 2019Few-Shot Video Classification via Temporal Alignmentcited by 124
- 2020Few-Shot Classification with Feature Map Reconstruction Networkscited by 116
- 2019Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulationcited by 103
- 2019Few-Shot Learning with Embedded Class Models and Shot-Free Meta Trainingcited by 57
- 2020Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Taskscited by 47
- 2021Learning Dynamic Alignment via Meta-filter for Few-shot Learningcited by 41
- 2020Learning to Learn Kernels with Variational Random Featurescited by 18
- 2020Learning to Learn Variational Semantic Memorycited by 17
- 2020Revisiting Meta-Learning as Supervised Learningcited by 14
- 2020Meta-Learning with Shared Amortized Variational Inferencecited by 14
- 2020Learning to Select Base Classes for Few-shot Classificationcited by 11
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.