Data to 7 Oct 2026

ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models

arXiv 2310.04564 ↗First posted 6 Oct 2023cs.LG cs.AIcited by 91

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained devices. Despite recent trends favoring alternative activation functions such as GELU or SiLU, known for increased computation, this study strongly advocates for reinstating ReLU activation in LLMs. We demonstrate that using the ReLU activation function has a negligible impact on convergence and performance while significantly reducing computation and weight transfer. This reduction is particularly valuable during the memory-bound inference step, where efficiency is paramount. Exploring sparsity patterns in ReLU-based LLMs, we unveil the reutilization of activated neurons for generating new tokens and leveraging these insights, we propose practical strategies to substantially reduce LLM inference computation up to three times, using ReLU activations with minimal performance trade-offs.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 69 of its 84 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

69 of 84 references.

References · 69

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2020Language Models are Few-Shot Learnerscited by 32,199
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2020Measuring Massive Multitask Language Understandingcited by 7,768
2020Scaling Laws for Neural Language Modelscited by 6,411
2016Layer Normalizationcited by 5,579
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2016Gaussian Error Linear Units (GELUs)cited by 3,431
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413

38 more

2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2016Pointer Sentinel Mixture Modelscited by 2,992
2022Training Compute-Optimal Large Language Modelscited by 2,927
2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2020GLU Variants Improve Transformercited by 1,724
2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
2019Root Mean Square Layer Normalizationcited by 1,425
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
2016Language Modeling with Gated Convolutional Networkscited by 688
2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
2017Self-Normalizing Neural Networkscited by 591
2019Improved Knowledge Distillation via Teacher Assistantcited by 567
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2022Efficiently Scaling Transformer Inferencecited by 529
2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
2023A Survey on Model Compression for Large Language Modelscited by 263
2023From Sparse to Soft Mixtures of Expertscited by 245
2023Speculative Decoding with Big Little Decodercited by 163
2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
2022A Review of Sparse Expert Models in Deep Learningcited by 147

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 33; 2025: 34; 2026: 21.

91 papers.

Sort:

By area:

2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
2024Training-Free Activation Sparsity in Large Language Modelscited by 51
2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 24

38 more

2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
2025Mixture of Experts Made Intrinsically Interpretablecited by 22
2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19
2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
2025Instruction-Following Pruning for Large Language Modelscited by 14
2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptationcited by 10
2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
2023Language Modeling on a SpiNNaker 2 Neuromorphic Chipcited by 6
2024Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODEcited by 6
2024Learning Neural Networks with Sparse Activationscited by 6
2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
2024Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neuronscited by 5
2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
2024Post-Training Statistical Calibration for Higher Activation Sparsitycited by 5
2025μnit Scaling: Simple and Scalable FP8 LLM Trainingcited by 5
2025Temporal Action Detection Model Compression by Progressive Block Dropcited by 5
2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 5
2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
2025Curvature Tuning: Provable Training-free Model Steering From a Single Parametercited by 4
2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
2026PowLU: An Activation Function for Stable Pre-Training of LLMscited by 4
2024The SkipSponge Attack: Sponge Weight Poisoning of Deep Neural Networkscited by 3

50 most cited of 91 All 91 (JSON)