mlsanityData to 14 Oct 2026

Truly Sparse Neural Networks at Scale

arXiv 2102.01732 ↗First posted 2 Feb 2021cs.LG cs.NEcited by 16

Not on the map. Why?

Abstract (arXiv)

Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks. Yet, this efficiency is just in theory. In practice, everyone uses a binary mask to simulate sparsity since the typical deep learning software and hardware are optimized for dense matrix operations. In this paper, we take an orthogonal approach, and we show that we can train truly sparse neural networks to harvest their full potential. To achieve this goal, we introduce three novel contributions, specially designed for sparse neural networks: (1) a parallel training algorithm and its corresponding sparse implementation from scratch, (2) an activation function with non-trainable parameters to favour the gradient flow, and (3) a hidden neurons importance metric to eliminate redundancies. All in one, we are able to break the record and to train the largest neural network ever trained in terms of representational power -- reaching the bat brain size. The results show that our approach has state-of-the-art performance while opening the path for an environmentally friendly artificial intelligence era.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 63 of its 106 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

63 of 106 references.

References · 63

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  12. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
38 more
  1. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  5. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  6. 2018Sparsified SGD with Memorycited by 376
  7. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  8. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  9. 2013Predicting Parameters in Deep Learningcited by 224
  10. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  11. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  12. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  13. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  14. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  15. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  16. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  17. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  18. 2020Sparse GPU Kernels for Deep Learningcited by 141
  19. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  20. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  21. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  22. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  23. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  24. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  25. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  26. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  27. 2018How Important Is a Neuron?cited by 73
  28. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 68
  29. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  30. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  31. 2020Sparse Weight Activation Trainingcited by 45
  32. 2016A topological insight into restricted Boltzmann machinescited by 32
  33. 2016Parallel SGD: When does averaging help?cited by 32
  34. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  35. 2020Growing Efficient Deep Networks by Structured Continuous Sparsificationcited by 28
  36. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  37. 2021Selfish Sparse RNN Trainingcited by 26
  38. 2019Cognitive Model Priors for Predicting Human Decisionscited by 25

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 4; 2025: 2; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  2. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 17
  3. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  4. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  5. 2022Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performancecited by 7
  6. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  7. 2022A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networkscited by 2
  8. 2024Nerva: a Truly Sparse Implementation of Neural Networkscited by 2
  9. 2023Learning Activation Functions for Sparse Neural Networkscited by 1
  10. 2023Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformerscited by 1
  11. 2024Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functionscited by 1
  12. 2024Are Sparse Neural Networks Better Hard Sample Learners?cited by 1
4 more
  1. 2025Sparse-to-Sparse Training of Diffusion Modelscited by 1
  2. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  3. 2026Motivation is Something You Needcited by 0
  4. 2026Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPUcited by 0