# Improving Generalization in Meta-Learning via Meta-Gradient Augmentation

arXiv: https://arxiv.org/abs/2306.08460 · HTML: https://mlsanity.com/paper/2306.08460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 66 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Task-Agnostic Meta-Learning for Few-shot Learning](https://mlsanity.com/paper/1805.07722/) (2018)
- [A Survey of Deep Meta-Learning](https://mlsanity.com/paper/2010.03522/) (2020)
- [Meta-Learning without Memorization](https://mlsanity.com/paper/1912.03820/) (2019)
- [The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models](https://mlsanity.com/paper/2012.06908/) (2020)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [On Mixup Regularization](https://mlsanity.com/paper/2006.06049/) (2020)
- [Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory](https://mlsanity.com/paper/1711.01244/) (2017)
- [Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy](https://mlsanity.com/paper/2004.00448/) (2020)
- [Meta-Learning Requires Meta-Augmentation](https://mlsanity.com/paper/2007.05549/) (2020)
- [Improving Generalization in Meta-learning via Task Augmentation](https://mlsanity.com/paper/2007.13040/) (2020)
- [La-MAML: Look-ahead Meta Learning for Continual Learning](https://mlsanity.com/paper/2007.13904/) (2020)
- [Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks](https://mlsanity.com/paper/2002.06753/) (2020)

## Abstract (arXiv)

Meta-learning methods typically follow a two-loop framework, where each loop potentially suffers from notorious overfitting, hindering rapid adaptation and generalization to new tasks. Existing schemes solve it by enhancing the mutual-exclusivity or diversity of training samples, but these data manipulation strategies are data-dependent and insufficiently flexible. This work alleviates overfitting in meta-learning from the perspective of gradient regularization and proposes a data-independent Meta-Gradient Augmentation (MGAug) method. The key idea is to first break the rote memories by network pruning to address memorization overfitting in the inner loop, and then the gradients of pruned sub-networks naturally form the high-quality augmentation of the meta-gradient to alleviate learner overfitting in the outer loop. Specifically, we explore three pruning strategies, including textitrandom width pruning, textitrandom parameter pruning, and a newly proposed textitcatfish pruning that measures a Meta-Memorization Carrying Amount (MMCA) score for each parameter and prunes high-score ones to break rote memories as much as possible. The proposed MGAug is theoretically guaranteed by the generalization bound from the PAC-Bayes framework. In addition, we extend a lightweight version, called MGAug-MaxUp, as a trade-off between performance gains and resource overhead. Extensive experiments on multiple few-shot learning benchmarks validate MGAug's effectiveness and significant improvement over various meta-baselines. The code is publicly available at urlhttps://github.com/xxLifeLover/Meta-Gradient-Augmentation.