mlsanityData to 7 Oct 2026

Scaling Laws for Sparsely-Connected Foundation Models

arXiv 2309.08520 ↗First posted 15 Sep 2023cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i.e., "foundation models"), in both vision and language domains. In this setting, we identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data, which we validate empirically across model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to characterize the "optimal sparsity", the sparsity level which yields the best performance for a given effective model size and training budget. For a fixed number of non-zero parameters, we identify that the optimal sparsity increases with the amount of data used for training. We also extend our study to different sparsity structures (such as the hardware-friendly n:m pattern) and strategies (such as starting from a pretrained dense model). Our findings shed light on the power and limitations of weight sparsity across various parameter and computational settings, offering both theoretical understanding and practical implications for leveraging sparsity towards computational efficiency improvements.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 53 of its 64 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

53 of 64 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2021Scaling Vision Transformerscited by 993
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  12. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  13. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  14. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  15. 2019The State of Sparsity in Deep Neural Networkscited by 471
  16. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  18. 2023Scaling Data-Constrained Language Modelscited by 391
  19. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  20. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  21. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  22. 2021Accelerating Sparse Deep Neural Networkscited by 213
  23. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  24. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  25. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  26. 2020Dynamic Model Pruning with Feedbackcited by 125
  27. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  28. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  29. 2022Broken Neural Scaling Lawscited by 102
  30. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  31. 2019Fast Sparse ConvNetscited by 86
  32. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  33. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  34. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  35. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  36. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  37. 2020On the Predictability of Pruning Across Scalescited by 33
  38. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 25

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 24; 2026: 1.

41 papers.

Sort:

By area:

  1. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  2. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  3. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  4. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  5. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  6. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  7. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  8. 2024Scaling Optimal LR Across Token Horizonscited by 35
  9. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  10. 2025Parallel Scaling Law for Language Modelscited by 29
  11. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
  12. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
29 more
  1. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  2. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  3. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  4. 2024Toward Inference-optimal Mixture-of-Expert Large Language Modelscited by 19
  5. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  6. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  7. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  8. 2024Effective Interplay between Sparsity and Quantization: From Theory to Practicecited by 16
  9. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  10. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  11. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Modelscited by 14
  12. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  13. 2024DAM: Towards A Foundation Model for Time Series Forecastingcited by 10
  14. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 10
  15. 2025Physics of Skill Learningcited by 8
  16. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7
  17. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  18. 2024Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipecited by 4
  19. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 4
  20. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  21. 2025Unified Scaling Laws for Compressed Representationscited by 3
  22. 2025The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Lawscited by 2
  23. 2025Complexity Scaling Laws for Neural Models using Combinatorial Optimizationcited by 1
  24. 2026Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layerscited by 1
  25. 2025Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validationcited by 0
  26. 2025How can representation dimension dominate structurally pruned LLMs?cited by 0
  27. 2025Hyperflux: Pruning Reveals Importancecited by 0
  28. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Modelscited by 0
  29. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Taskscited by 0