mlsanityData to 7 Oct 2026

ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs

arXiv 2402.03804 ↗First posted 6 Feb 2024cs.LG cs.AIcited by 66

Not on the map. Why?

Abstract (arXiv)

Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neurons. While traditional approaches focus on ReLU-based LLMs, leveraging zeros in activation values, we broaden the scope of sparse LLMs beyond zero activation values. We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold, demonstrating that non-ReLU LLMs also exhibit sparse activation. To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU2. The results indicate that models employing ReLU2 excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs. We will release the code to facilitate future research.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 63 of its 76 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

63 of 76 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2020GLU Variants Improve Transformercited by 1,724
  14. 2020Reformer: The Efficient Transformercited by 1,658
  15. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  16. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  17. 2019Root Mean Square Layer Normalizationcited by 1,425
  18. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  19. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  20. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  21. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  22. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  23. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  24. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  25. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  26. 2019Are Sixteen Heads Really Better than One?cited by 803
  27. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  28. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  29. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  30. 2022Efficiently Scaling Transformer Inferencecited by 529
  31. 2023The Falcon Series of Open Language Modelscited by 526
  32. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  33. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  34. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  35. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  36. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  37. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  38. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 25; 2026: 24.

66 papers.

Sort:

By area:

  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  2. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  3. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  4. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  5. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  6. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  7. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  8. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  9. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  10. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  11. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
  12. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
38 more
  1. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  2. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  3. 2024Yi-Lightning Technical Reportcited by 11
  4. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  5. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  6. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  7. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  8. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  9. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  10. 2026Simply Stabilizing the Loop via Fully Looped Transformercited by 6
  11. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 5
  12. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  13. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  14. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
  15. 2025iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± icited by 5
  16. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  17. 2024Achieving Sparse Activation in Small Language Modelscited by 4
  18. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  19. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  20. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 4
  21. 2024Dynamic Activation Pitfalls in LLaMA Models: An Empirical Studycited by 3
  22. 2024How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neuronscited by 3
  23. 2025Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsitycited by 3
  24. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 3
  25. 2026Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Designcited by 3
  26. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  27. 2026More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activationscited by 3
  28. 2024Mixture of Hidden-Dimensions Transformercited by 2
  29. 2025Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapterscited by 2
  30. 2024Activation Sparsity Opportunities for Compressing General Large Language Modelscited by 1
  31. 2025COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projectioncited by 1
  32. 2025Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoningcited by 1
  33. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  34. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  35. 2026How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNscited by 1
  36. 2026ActTail: Global Activation Sparsity in Large Language Modelscited by 1
  37. 2026Bug or Feature2: Weight Drift, Activation Sparsity and Spikescited by 1
  38. 2024Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Modelscited by 0

50 most cited of 66 All 66 (JSON)