mlsanityData to 7 Oct 2026

Sparse Training Theory for Scalable and Efficient Agents

arXiv 2103.01636 ↗First posted 2 Mar 2021cs.AI cs.LG cs.MAcited by 12

Not on the map. Why?

Abstract (arXiv)

A fundamental task for artificial intelligence is learning. Deep Neural Networks have proven to cope perfectly with all learning paradigms, i.e. supervised, unsupervised, and reinforcement learning. Nevertheless, traditional deep learning approaches make use of cloud computing facilities and do not scale well to autonomous agents with low computational resources. Even in the cloud, they suffer from computational and memory limitations, and they cannot be used to model adequately large physical worlds for agents which assume networks with billions of neurons. These issues are addressed in the last few years by the emerging topic of sparse training, which trains sparse networks from scratch. This paper discusses sparse training state-of-the-art, its challenges and limitations while introducing a couple of new theoretical research directions which has the potential of alleviating sparse training limitations to push deep learning scalability well beyond its current boundaries. Nevertheless, the theoretical advancements impact in complex multi-agents settings is discussed from a real-world perspective, using the smart grid case study.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

34 of 53 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  5. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  10. 2020Supermasks in Superpositioncited by 223
  11. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  12. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
22 more
  1. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  2. 2020The Hardware Lotterycited by 160
  3. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  4. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  5. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  6. 2020SpaceNet: Make Free Space For Continual Learningcited by 53
  7. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  8. 2018Multi-objective Evolutionary Federated Learningcited by 42
  9. 2016A topological insight into restricted Boltzmann machinescited by 32
  10. 2017On-line Building Energy Optimization using Deep Reinforcement Learningcited by 27
  11. 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 27
  12. 2021Selfish Sparse RNN Trainingcited by 26
  13. 2019Cognitive Model Priors for Predicting Human Decisionscited by 24
  14. 2017Deep Expander Networks: Efficient Deep Networks from Graph Theorycited by 23
  15. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  16. 2020Topological Insights into Sparse Neural Networkscited by 21
  17. 2019One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluationcited by 14
  18. 2019RadiX-Net: Structured Sparse Matrices for Deep Neural Networkscited by 12
  19. 2018Pre-Defined Sparse Neural Networks with Hardware Accelerationcited by 9
  20. 2020Sparse Linear Networks with a Fixed Butterfly Structure: Theory and Practicecited by 9
  21. 2019Sparse Deep Neural Network Graph Challengecited by 6
  22. 2017Complexity of Scheduling Charging in the Smart Gridcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 2; 2023: 6; 2024: 1.

12 papers.

  1. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  2. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  3. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  4. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
  5. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  6. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  7. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  8. 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
  9. 2023Training Large Language Models Efficiently with Sparsity and Dataflowcited by 7
  10. 2024Unveiling the Power of Sparse Neural Networks for Feature Selectioncited by 3
  11. 2023Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformerscited by 1
  12. 2023Feather: An Elegant Solution to Effective DNN Sparsificationcited by 1