mlsanityData to 7 Oct 2026

GradMax: Growing Neural Networks using Gradient Information

arXiv 2201.05125 ↗First posted 13 Jan 2022cs.LG cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights and find the optimal initialization efficiently by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

32 of 52 references.

References · 32

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2018Rethinking the Value of Network Pruningcited by 595
  6. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  7. 2017Large-Scale Evolution of Image Classifierscited by 348
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  10. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  11. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
20 more
  1. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  2. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  3. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  4. 2016Network Morphismcited by 63
  5. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  6. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  7. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  8. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  9. 2020Towards Learning Convolutions from Scratchcited by 46
  10. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  11. 2017Parallel Multi Channel Convolution using General Matrix Multiplicationcited by 30
  12. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  13. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26
  14. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  15. 2019Finding the Needle in the Haystack with Convolutions: on the benefits of architectural biascited by 19
  16. 2019Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descentcited by 17
  17. 2020Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splittingcited by 14
  18. 2017Efficient Architecture Search by Network Transformationcited by 9
  19. 2018CompNet: Neural networks growing via the compact network morphismcited by 2
  20. 2019No Pressure! Addressing the Problem of Local Minima in Manifold Learning Algorithmscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 15; 2024: 14; 2025: 14; 2026: 14.

64 papers.

Sort:

By area:

  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  3. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  4. 2022Staged Training for Transformer Language Modelscited by 49
  5. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  6. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  7. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  8. 2022When, where, and how to add new neurons to ANNscited by 18
  9. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  10. 2022Streamable Neural Fieldscited by 13
  11. 2023Self-Expanding Neural Networkscited by 13
  12. 2025Steepest Descent Density Control for Compact 3D Gaussian Splattingcited by 13
38 more
  1. 2023Composable Function-preserving Expansions for Transformer Architecturescited by 11
  2. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  3. 2024Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimallycited by 9
  4. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  5. 2023Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptationcited by 8
  6. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  7. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  8. 2024Directed Structural Adaptation to Overcome Statistical Conflicts and Enable Continual Learningcited by 4
  9. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  10. 2024When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networkscited by 3
  11. 2024Towards Optimal Adapter Placement for Efficient Transfer Learningcited by 3
  12. 2024Learning Fine-grained Parameter Sharing via Sparse Tensor Decompositioncited by 3
  13. 2022Noisy Heuristics NAS: A Network Morphism based Neural Architecture Search using Heuristicscited by 2
  14. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  15. 2023AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networkscited by 2
  16. 2023Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removalcited by 2
  17. 2025Growth strategies for arbitrary DAG neural architecturescited by 2
  18. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2
  19. 2025Foundations of a Developmental Design Paradigm for Integrated Continual Learning, Deliberative Behavior, and Comprehensibilitycited by 2
  20. 2025Reusing Overtrained Language Models Saturates Scalingcited by 2
  21. 2026Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphscited by 2
  22. 2022Pretraining a Neural Network before Knowing Its Architecturecited by 1
  23. 2023STen: Productive and Efficient Sparsity in PyTorchcited by 1
  24. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
  25. 2024Level Set Teleportation: An Optimization Perspectivecited by 1
  26. 2024Early Period of Training Impacts Adaptation for Out-of-Distribution Generalization: An Empirical Studycited by 1
  27. 2024Quantum Circuit Training with Growth-Based Architecturescited by 1
  28. 2025The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participationcited by 1
  29. 2025Growing with Experience: Growing Neural Networks in Deep Reinforcement Learningcited by 1
  30. 2025Shared-Weights Extender and Gradient Voting for Neural Network Expansioncited by 1
  31. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  32. 2023Adaptive Neural Networks Using Residual Fittingcited by 0
  33. 2024Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architecturescited by 0
  34. 2024LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameterscited by 0
  35. 2024Growing Efficient Accurate and Robust Neural Networks on the Edgecited by 0
  36. 2025A Guaranteed-Stable Neural Network Approach for Optimal Control of Nonlinear Systemscited by 0
  37. 2025Evolutionary Developmental Biology Can Serve as the Conceptual Foundation for a New Design Paradigm in Artificial Intelligencecited by 0
  38. 2025Data Classification with Dynamically Growing and Shrinking Neural Networkscited by 0

50 most cited of 64 All 64 (JSON)