mlsanityData to 14 Oct 2026

Learning where to learn: Gradient sparsity in meta and continual learning

arXiv 2110.14402 ↗First posted 27 Oct 2021cs.LG cs.NEcited by 33

Not on the map. Why?

Abstract (arXiv)

Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 40 of its 68 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

40 of 68 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  8. 2016Matching Networks for One Shot Learningcited by 2,255
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  12. 2019A Closer Look at Few-shot Classificationcited by 684
28 more
  1. 2019Meta-Learning with Implicit Gradientscited by 525
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  3. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  4. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  5. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 379
  6. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  7. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  8. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  9. 2018How to train your MAMLcited by 309
  10. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  11. 2019Recurrent Independent Mechanismscited by 219
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  13. 2019Meta-Learning Representations for Continual Learningcited by 189
  14. 2019Meta-Learning with Warped Gradient Descentcited by 98
  15. 2019Meta-Curvaturecited by 70
  16. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 69
  17. 2017FreezeOut: Accelerate Training by Progressively Freezing Layerscited by 54
  18. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  19. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  20. 2020La-MAML: Look-ahead Meta Learning for Continual Learningcited by 52
  21. 2020TaskNorm: Rethinking Batch Normalization for Meta-Learningcited by 51
  22. 2021Fast and Slow Learning of Recurrent Independent Mechanismscited by 36
  23. 2020Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updatescited by 34
  24. 2019Modular Meta-Learning with Shrinkagecited by 31
  25. 2018Been There, Done That: Meta-Learning with Episodic Recallcited by 26
  26. 2021A contrastive rule for meta-learningcited by 14
  27. 2019Meta-descent for Online, Continual Predictioncited by 10
  28. 2019Learning Feature Relevance Through Step Size Adaptation in Temporal-Difference Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 14; 2023: 7; 2024: 6; 2025: 4; 2026: 1.

33 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 732
  2. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  3. 2021Continual Learning via Local Module Compositioncited by 63
  4. 2023MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learningcited by 40
  5. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  6. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  7. 2022Meta-Learning Sparse Compression Networkscited by 22
  8. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  9. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  10. 2023When Meta-Learning Meets Online and Continual Learning: A Surveycited by 12
  11. 2023Meta-Learning with a Geometry-Adaptive Preconditionercited by 11
  12. 2023Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapterscited by 10
21 more
  1. 2022Meta-Learning with Self-Improving Momentum Targetcited by 9
  2. 2022Meta-Learning via Classifier(-free) Diffusion Guidancecited by 9
  3. 2023Online Continual Learning for Robust Indoor Object Recognitioncited by 9
  4. 2022p-Meta: Towards On-device Deep Model Adaptationcited by 8
  5. 2022Continual Feature Selection: Spurious Features in Continual Learningcited by 7
  6. 2022Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptilecited by 6
  7. 2022Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computingcited by 6
  8. 2022Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networkscited by 4
  9. 2024Enhancing Few-Shot Image Classification through Learnable Multi-Scale Embedding and Attention Mechanismscited by 4
  10. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Expertscited by 3
  11. 2023Competitive plasticity to reduce the energetic costs of learningcited by 1
  12. 2024Learning to Learn without Forgetting using Attentioncited by 1
  13. 2024Task-Specific Preconditioner for Cross-Domain Few-Shot Learningcited by 1
  14. 2025Meta-Continual Learning of Neural Fieldscited by 1
  15. 2025MoEMeta: Mixture-of-Experts Meta Learning for Few-Shot Relational Learningcited by 1
  16. 2026Group and Exclusive Sparse Regularization-based Continual Learning of CNNscited by 1
  17. 2022New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 0
  18. 2024Seq-to-Final: A Benchmark for Tuning from Sequential Distributions to a Final Time Pointcited by 0
  19. 2024Prototype Optimization with Neural ODE for Few-Shot Learningcited by 0
  20. 2025Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Expertscited by 0
  21. 2025Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learningcited by 0