mlsanityData to 7 Oct 2026

Deep Rewiring: Training very sparse deep networks

arXiv 1711.05136 ↗First posted 14 Nov 2017cs.NE cs.AI cs.DCcited by 136

Not on the map. Why?

Abstract (arXiv)

Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 26; 2023: 28; 2024: 22; 2025: 15; 2026: 9.

136 papers.

Sort:

By area:

  1. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  2. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  3. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  4. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  5. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  6. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  7. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  8. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  9. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  10. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  11. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  12. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
38 more
  1. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  2. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  3. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  4. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  5. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  6. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  7. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  8. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  9. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  10. 2021Pruning of Deep Spiking Neural Networks through Gradient Rewiringcited by 41
  11. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  12. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  13. 2021A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardwarecited by 32
  14. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  15. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  16. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  17. 2022Dual Lottery Ticket Hypothesiscited by 27
  18. 2021Selfish Sparse RNN Trainingcited by 26
  19. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  20. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 25
  21. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  22. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  23. 2023LEMON: Lossless model expansioncited by 23
  24. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  25. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  26. 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
  27. 2021RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Mergingcited by 19
  28. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  29. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 17
  30. 2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
  31. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  32. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  33. 2021Attention-based Convolutional Autoencoders for 3D-Variational Data Assimilationcited by 15
  34. 2021Truly Sparse Neural Networks at Scalecited by 15
  35. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  36. 2023Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networkscited by 15
  37. 2024LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernelscited by 15
  38. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13

50 most cited of 136 All 136 (JSON)