mlsanityData to 7 Oct 2026

Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science

arXiv 1707.04780 ↗First posted 15 Jul 2017cs.NE cs.AI cs.LGcited by 325

Not on the map. Why?

Abstract (arXiv)

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (ErdHos-R'enyi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

Key facts. Cited by 325 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 55; 2022: 60; 2023: 67; 2024: 60; 2025: 54; 2026: 29.

325 papers.

Sort:

By area:

  1. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  2. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  6. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  7. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  8. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  9. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  10. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  11. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  12. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
38 more
  1. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  2. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  3. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  4. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  5. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  6. 2022Deep Differentiable Logic Gate Networkscited by 67
  7. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  8. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  9. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  10. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  11. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  12. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  13. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  14. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  15. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  16. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  17. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  18. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  19. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  20. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  21. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  22. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  23. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  24. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  25. 2025Weight-sparse transformers have interpretable circuitscited by 40
  26. 2022Machine Learning in Aerodynamic Shape Optimizationcited by 39
  27. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  28. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  29. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  30. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  31. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  32. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  33. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 35
  34. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  35. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  36. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  37. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  38. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29

50 most cited of 325 All 325 (JSON)