mlsanityData to 7 Oct 2026

Towards Structured Dynamic Sparse Pre-Training of BERT

arXiv 2108.06277 ↗First posted 13 Aug 2021cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Identifying algorithms for computational efficient unsupervised training of large language models is an important and active area of research. In this work, we develop and study a straightforward, dynamic always-sparse pre-training approach for BERT language modeling task, which leverages periodic compression steps based on magnitude pruning followed by random parameter re-allocation. This approach enables us to achieve Pareto improvements in terms of the number of floating-point operations (FLOPs) over statically sparse and dense models across a broad spectrum of network sizes. Furthermore, we demonstrate that training remains FLOP-efficient when using coarse-grained block sparsity, making it particularly promising for efficient execution on modern hardware accelerators.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 51 of its 59 references that are in the map. First posted on arXiv 13 Aug 2021.

Builds on

51 of 59 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  10. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  12. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
38 more
  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  4. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  5. 2019The State of Sparsity in Deep Neural Networkscited by 471
  6. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  9. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  10. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  12. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  13. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  14. 2019Structured Pruning of Large Language Modelscited by 201
  15. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  16. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  17. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  18. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  19. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  20. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  21. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  22. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  23. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  24. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  25. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  26. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  27. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  28. 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
  29. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  30. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  31. 2017Block-Sparse Recurrent Neural Networkscited by 38
  32. 2020A Gradient Flow Framework For Analyzing Network Pruningcited by 35
  33. 2021Selfish Sparse RNN Trainingcited by 26
  34. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  35. 2017Learning Intrinsic Sparse Structures within Long Short-Term Memorycited by 18
  36. 2019One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluationcited by 14
  37. 2019Group Pruning using a Bounded-Lp norm for Group Gating and Regularizationcited by 3
  38. 2020Ramanujan Bipartite Graph Products for Efficient Block Sparse Neural Networkscited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 7; 2024: 3.

12 papers.

  1. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  2. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  3. 2022Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performancecited by 7
  4. 2022Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Trainingcited by 6
  5. 2023Calibrating the Rigged Lottery: Making All Tickets Reliablecited by 4
  6. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  7. 2023PopSparse: Accelerated block sparse matrix multiplication on IPUcited by 2
  8. 2023REST: Enhancing Group Robustness in DNNs through Reweighted Sparse Trainingcited by 2
  9. 2023Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformerscited by 1
  10. 2023Reducing Memory Requirements for the IPU using Butterfly Factorizationscited by 0
  11. 2024MediSwift: Efficient Sparse Pre-trained Biomedical Language Modelscited by 0
  12. 2024Mixed Sparsity Training: Achieving 4× FLOP Reduction for Transformer Pretrainingcited by 0