mlsanityData to 14 Oct 2026

Sparse is Enough in Scaling Transformers

arXiv 2111.12763 ↗First posted 24 Nov 2021cs.LG cs.CLcited by 86

Not on the map. Why?

Abstract (arXiv)

Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach. We address this problem by leveraging sparsity. We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size. Surprisingly, the sparse layers are enough to obtain the same perplexity as the standard Transformer with the same number of parameters. We also integrate with prior sparsity approaches to attention and enable fast inference on long sequences even with limited memory. This results in performance competitive to the state-of-the-art on long text summarization.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

40 of 47 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
28 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  6. 2020Efficient Transformers: A Surveycited by 790
  7. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  8. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  11. 2019The State of Sparsity in Deep Neural Networkscited by 476
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  14. 2020Sparse Sinkhorn Attentioncited by 251
  15. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  16. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  17. 2019Adaptive Attention Span in Transformerscited by 149
  18. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  19. 2015Neural GPUs Learn Algorithmscited by 121
  20. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  21. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  22. 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
  23. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  24. 2019Fully Quantized Transformer for Machine Translationcited by 36
  25. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  26. 2018Discrete Autoencoders for Sequence Modelscited by 24
  27. 2020Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecturecited by 23
  28. 2019Flexible Multi-task Networks by Learning Parameter Allocationcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 18; 2023: 29; 2024: 24; 2025: 14.

86 papers.

Sort:

By area:

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2022Efficiently Scaling Transformer Inferencecited by 539
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  5. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  6. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  9. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  10. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  11. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  12. 2023Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Optcited by 81
38 more
  1. 2024All-in-one simulation-based inferencecited by 74
  2. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  3. 2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
  4. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  5. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  6. 2023Dynamic Sparse Training with Structured Sparsitycited by 38
  7. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  8. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  9. 2024Multi-Candidate Speculative Decodingcited by 31
  10. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  11. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 28
  12. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  13. 2024Selective Attention Improves Transformercited by 26
  14. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  15. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 19
  16. 2022Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworkscited by 18
  17. 2023Beyond Classical Attention: Quantum Attention for Scalable Computationcited by 18
  18. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 18
  19. 2024Attention layers provably solve single-location regressioncited by 17
  20. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 16
  21. 2023MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentationcited by 14
  22. 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
  23. 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
  24. 2022Sparse Mixers: Combining MoE and Mixing to build a more efficient BERTcited by 12
  25. 2024CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Informationcited by 11
  26. 2024Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limitscited by 11
  27. 2024Conditional computation in neural networks: principles and research trendscited by 10
  28. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9
  29. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Modelcited by 9
  30. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  31. 2022Efficient Language Modeling with Sparse all-MLPcited by 8
  32. 2023Sparse Binary Transformers for Multivariate Time Series Modelingcited by 8
  33. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  34. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  35. 2023MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learningcited by 7
  36. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  37. 2022A Theoretical View on Sparsely Activated Networkscited by 6
  38. 2022Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learningcited by 5

50 most cited of 86 All 86 (JSON)