# Meta-Learning with Adaptive Hyperparameters

arXiv: https://arxiv.org/abs/2011.00209 · HTML: https://mlsanity.com/paper/2011.00209/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2020.

## Built on by (most cited first, 30 of 79)

- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Auto-Lambda: Disentangling Dynamic Task Relationships](https://mlsanity.com/paper/2202.03091/) (2022)
- [Rectifying the Shortcut Learning of Background for Few-Shot Learning](https://mlsanity.com/paper/2107.07746/) (2021)
- [Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning](https://mlsanity.com/paper/2110.03909/) (2021)
- [A Closer Look at Few-shot Classification Again](https://mlsanity.com/paper/2301.12246/) (2023)
- [MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning](https://mlsanity.com/paper/2307.16424/) (2023)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners](https://mlsanity.com/paper/2412.17256/) (2024)
- [Meta Knowledge Distillation](https://mlsanity.com/paper/2202.07940/) (2022)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy](https://mlsanity.com/paper/2406.16087/) (2024)
- [Data-Efficient Brain Connectome Analysis via Multi-Task Meta-Learning](https://mlsanity.com/paper/2206.04486/) (2022)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2301.01156/) (2023)
- [MetaNODE: Prototype Optimization as a Neural ODE for Few-Shot Learning](https://mlsanity.com/paper/2103.14341/) (2021)
- [Meta-Learning with a Geometry-Adaptive Preconditioner](https://mlsanity.com/paper/2304.01552/) (2023)
- [LFS-GAN: Lifelong Few-Shot Image Generation](https://mlsanity.com/paper/2308.11917/) (2023)
- [Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2411.01432/) (2024)
- [Multidimensional Belief Quantification for Label-Efficient Meta-Learning](https://mlsanity.com/paper/2203.12768/) (2022)
- [Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classification](https://mlsanity.com/paper/2207.00784/) (2022)
- [Dataset Distillation via Adversarial Prediction Matching](https://mlsanity.com/paper/2312.08912/) (2023)
- [Revisiting Local Descriptor for Improved Few-Shot Classification](https://mlsanity.com/paper/2103.16009/) (2021)
- [Stateless Neural Meta-Learning using Second-Order Gradients](https://mlsanity.com/paper/2104.10527/) (2021)
- [Prototype Completion for Few-Shot Learning](https://mlsanity.com/paper/2108.05010/) (2021)
- [Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2208.01573/) (2022)
- [MetaCloth: Learning Unseen Tasks of Dense Fashion Landmark Detection from a Few Samples](https://mlsanity.com/paper/2112.02763/) (2021)
- [ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning](https://mlsanity.com/paper/2109.13305/) (2021)
- [SGMNet: Scene Graph Matching Network for Few-Shot Remote Sensing Scene Classification](https://mlsanity.com/paper/2110.04494/) (2021)
- [Contextual Gradient Scaling for Few-Shot Learning](https://mlsanity.com/paper/2110.10353/) (2021)

## Abstract (arXiv)

Despite its popularity, several recent works question the effectiveness of MAML when test tasks are different from training tasks, thus suggesting various task-conditioned methodology to improve the initialization. Instead of searching for better task-aware initialization, we focus on a complementary factor in MAML framework, inner-loop optimization (or fast adaptation). Consequently, we propose a new weight update rule that greatly enhances the fast adaptation process. Specifically, we introduce a small meta-network that can adaptively generate per-step hyperparameters: learning rate and weight decay coefficients. The experimental results validate that the Adaptive Learning of hyperparameters for Fast Adaptation (ALFA) is the equally important ingredient that was often neglected in the recent few-shot learning approaches. Surprisingly, fast adaptation from random initialization with ALFA can already outperform MAML.