# Regularizing Meta-Learning via Gradient Dropout

arXiv: https://arxiv.org/abs/2004.05859 · HTML: https://mlsanity.com/paper/2004.05859/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2020.

## Built on by (most cited first, 15 of 15)

- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [C-Mixup: Improving Generalization in Regression](https://mlsanity.com/paper/2210.05775/) (2022)
- [Meta-Learning with Fewer Tasks through Task Interpolation](https://mlsanity.com/paper/2106.02695/) (2021)
- [Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models](https://mlsanity.com/paper/2203.01104/) (2022)
- [Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models](https://mlsanity.com/paper/2203.16755/) (2022)
- [DropKAN: Regularizing KANs by masking post-activations](https://mlsanity.com/paper/2407.13044/) (2024)
- [Meta-Learned Invariant Risk Minimization](https://mlsanity.com/paper/2103.12947/) (2021)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [A Channel Coding Benchmark for Meta-Learning](https://mlsanity.com/paper/2107.07579/) (2021)
- [Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence](https://mlsanity.com/paper/2603.05960/) (2026)
- [B-SMALL: A Bayesian Neural Network approach to Sparse Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2101.00203/) (2021)
- [Learning to generate imaginary tasks for improving generalization in meta-learning](https://mlsanity.com/paper/2206.04335/) (2022)
- [Improving Generalization in Meta-Learning via Meta-Gradient Augmentation](https://mlsanity.com/paper/2306.08460/) (2023)
- [Neuromodulated Meta-Learning](https://mlsanity.com/paper/2411.06746/) (2024)
- [Game-Theoretic Gradient Control for Robust Neural Network Training](https://mlsanity.com/paper/2507.19143/) (2025)

## Abstract (arXiv)

With the growing attention on learning-to-learn new tasks using only a few examples, meta-learning has been widely used in numerous problems such as few-shot classification, reinforcement learning, and domain generalization. However, meta-learning models are prone to overfitting when there are no sufficient training tasks for the meta-learners to generalize. Although existing approaches such as Dropout are widely used to address the overfitting problem, these methods are typically designed for regularizing models of a single task in supervised training. In this paper, we introduce a simple yet effective method to alleviate the risk of overfitting for gradient-based meta-learning. Specifically, during the gradient-based adaptation stage, we randomly drop the gradient in the inner-loop optimization of each parameter in deep neural networks, such that the augmented gradients improve generalization to new tasks. We present a general form of the proposed gradient dropout regularization and show that this term can be sampled from either the Bernoulli or Gaussian distribution. To validate the proposed method, we conduct extensive experiments and analysis on numerous computer vision tasks, demonstrating that the gradient dropout regularization mitigates the overfitting problem and improves the performance upon various gradient-based meta-learning frameworks.