mlsanityData to 7 Oct 2026

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

arXiv 2406.05955 ↗First posted 10 Jun 2024cs.LG cs.CLcited by 37

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Exploiting activation sparsity is a promising approach to significantly accelerating the inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. Simply replacing these functions with ReLU fails to achieve sufficient sparsity. Moreover, inadequate training data can further increase the risk of performance degradation. To address these challenges, we propose a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. Additionally, we leverage sparse activation patterns within the Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to further boost efficiency. By applying our neuron sparsification method to the Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are activated per inference iteration, respectively, while achieving even more powerful model performance. Evaluation results demonstrate that this sparsity achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second. Our models are available at urlhttps://huggingface.co/PowerInfer

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 48 of its 60 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

48 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
33 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  13. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  14. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  15. 2019Are Sixteen Heads Really Better than One?cited by 803
  16. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  17. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  18. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  19. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  20. 2024InternLM2 Technical Reportcited by 391
  21. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  22. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  23. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  24. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  25. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  26. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  27. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  28. 2013Deep Learning of Representations: Looking Forwardcited by 84
  29. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  30. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  31. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  32. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  33. 2024FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inferencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 16; 2026: 8.

37 papers.

Sort:

By area:

  1. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  4. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 24
  5. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  6. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  7. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  8. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  9. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  10. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16
  11. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  12. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
25 more
  1. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  2. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  3. 2024Post-Training Statistical Calibration for Higher Activation Sparsitycited by 5
  4. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  5. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
  6. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  7. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  8. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  9. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 4
  10. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3
  11. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 3
  12. 2025SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Modelscited by 2
  13. 2024RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devicescited by 1
  14. 2025COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projectioncited by 1
  15. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Modelscited by 1
  16. 2026Bug or Feature2: Weight Drift, Activation Sparsity and Spikescited by 1
  17. 2026Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendationcited by 1
  18. 2025Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approachescited by 0
  19. 2025VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunkingcited by 0
  20. 2026WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsitycited by 0
  21. 2026SVD Contextual Sparsity Predictors for Fast LLM Inferencecited by 0
  22. 2026Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMscited by 0
  23. 2026NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetchingcited by 0
  24. 2026Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardwarecited by 0
  25. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
  2. 2024Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment
  3. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference
  4. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  5. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  6. 2026ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
  7. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
  8. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference