mlsanityData to 7 Oct 2026

Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks

arXiv 2102.08574 ↗First posted 17 Feb 2021cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

23 of 25 references.

References · 23

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016Learning without Forgettingcited by 2,465
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2016Progressive Neural Networkscited by 1,332
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  10. 2017Progressive Neural Architecture Searchcited by 483
  11. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  12. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
11 more
  1. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  2. 2018Reinforced Continual Learningcited by 186
  3. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  4. 2017Incremental Learning Through Deep Adaptationcited by 108
  5. 2016Network Morphismcited by 63
  6. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  7. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  8. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26
  9. 2019Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descentcited by 17
  10. 2019Efficient Forward Architecture Searchcited by 14
  11. 2020Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splittingcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 7; 2024: 8; 2025: 10; 2026: 10.

48 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2022Continual Learning and Private Unlearningcited by 124
  3. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  4. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  5. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  6. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  7. 2022Residual Mixture of Expertscited by 39
  8. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 29
  9. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  10. 2021On Anytime Learning at Macroscalecited by 20
  11. 2022When, where, and how to add new neurons to ANNscited by 18
  12. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
36 more
  1. 2022Streamable Neural Fieldscited by 13
  2. 2023Self-Expanding Neural Networkscited by 13
  3. 2025Steepest Descent Density Control for Compact 3D Gaussian Splattingcited by 13
  4. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  5. 2024Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimallycited by 9
  6. 2023Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptationcited by 8
  7. 2023MixtureGrowth: Growing Neural Networks by Recombining Learned Parameterscited by 7
  8. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  9. 2025Adaptive Width Neural Networkscited by 4
  10. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  11. 2024When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networkscited by 3
  12. 2025Growth strategies for arbitrary DAG neural architecturescited by 2
  13. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2
  14. 2025Expand Neurons, Not Parameterscited by 2
  15. 2026Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generationcited by 2
  16. 2022Incremental Learning with Differentiable Architecture and Forgetting Searchcited by 1
  17. 2023SensLI: Sensitivity-Based Layer Insertion for Neural Networkscited by 1
  18. 2024Learning Morphisms with Gauss-Newton Approximation for Growing Networkscited by 1
  19. 2025Variational Kolmogorov-Arnold Networkcited by 1
  20. 2025Shared-Weights Extender and Gradient Voting for Neural Network Expansioncited by 1
  21. 2021A Meta-Learned Neuron model for Continual Learningcited by 0
  22. 2022Growing Neural Network with Shared Parametercited by 0
  23. 2024Beyond Uniform Scaling: Exploring Depth Heterogeneity in Neural Architecturescited by 0
  24. 2024Growing Deep Neural Network Considering with Similarity between Neuronscited by 0
  25. 2024Growing Efficient Accurate and Robust Neural Networks on the Edgecited by 0
  26. 2025A Guaranteed-Stable Neural Network Approach for Optimal Control of Nonlinear Systemscited by 0
  27. 2025Long-Tailed Data Classification by Increasing and Decreasing Neurons During Trainingcited by 0
  28. 2025Gradual Capacity Growth for Sparse Network Discoverycited by 0
  29. 2026Growing Networks with Autonomous Pruningcited by 0
  30. 2026On the Stability of Growth in Structural Plasticity: Forward-Active yet Backward-Starvedcited by 0
  31. 2026Recursive Block-Diagonal Coupling for Resource-Efficient Training of Vision Modelscited by 0
  32. 2026An optimal control approach for neural network architecture adaptation with a posteriori error estimationcited by 0
  33. 2026Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterioncited by 0
  34. 2026Across the Loss Landscape with Progressive Growthcited by 0
  35. 2026Neuron-Level Architecture Growth: A Controlled Evaluation for EEG Time-Series Decodingcited by 0
  36. 2026CAGE-NAS: Certified Functional Descent for Efficient Model Growthcited by 0