mlsanityData to 7 Oct 2026

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

arXiv 2210.06313 ↗First posted 12 Oct 2022cs.LG cs.CL cs.CVcited by 121

Not on the map. Why?

Abstract (arXiv)

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by sparse we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 76 of its 118 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

76 of 118 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2017Billion-scale similarity search with GPUscited by 2,477
38 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2021Going deeper with Image Transformerscited by 759
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 441
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  13. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  14. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  15. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  16. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  17. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  18. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  19. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  20. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  21. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  22. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  23. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  24. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  25. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  26. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  27. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  28. 2021Sparse is Enough in Scaling Transformerscited by 86
  29. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  30. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  31. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  32. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  33. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 71
  34. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  35. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  36. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  37. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  38. 2016Convolutional Neural Networks Analyzed via Convolutional Sparse Codingcited by 51

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 52; 2025: 30; 2026: 22.

121 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  4. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  5. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  6. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  7. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  8. 2023White-Box Transformers via Sparse Rate Reductioncited by 77
  9. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  10. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  11. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  12. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
38 more
  1. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  2. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  3. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  4. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  5. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  6. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  7. 2024A Survey of Small Language Modelscited by 31
  8. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  9. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  10. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  11. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  12. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  13. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  14. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19
  15. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  16. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  17. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  18. 2024Merging Text Transformer Models from Different Initializationscited by 16
  19. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15
  20. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  21. 2023Accelerating Deep Neural Networks via Semi-Structured Activation Sparsitycited by 10
  22. 2024Tandem Transformers for Inference Efficient LLMscited by 10
  23. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  24. 2024Revealing the Parallel Multilingual Learning within Large Language Modelscited by 8
  25. 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
  26. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  27. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  28. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
  29. 2024Progress Measures for Grokking on Real-world Taskscited by 7
  30. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  31. 2024Event-based Optical Flow on Neuromorphic Processor: ANN vs. SNN Comparison based on Activation Sparsificationcited by 7
  32. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillationcited by 7
  33. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 7
  34. 2023Language Modeling on a SpiNNaker 2 Neuromorphic Chipcited by 6
  35. 2024Learning Neural Networks with Sparse Activationscited by 6
  36. 2024First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Modelscited by 6
  37. 2024CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsificationcited by 6
  38. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6

50 most cited of 121 All 121 (JSON)