mlsanityData to 14 Oct 2026

DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs

arXiv 2506.20194 ↗First posted 25 Jun 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these demands, most methods ignore activation sparsity observed at runtime. We reinterpret activation sparsity as dynamic structured weight sparsity and propose DuoGPT, a unified framework that constructs dual-sparse (spMspV) workloads by combining unstructured weight pruning with activation sparsity. To preserve accuracy, we extend the Optimal Brain Compression (OBC) framework with activation-aware calibration and introduce output residuals from the dense model as correction terms. We further optimize the solution for efficient GPU execution, enabling scalability to billion-parameter LLMs. Evaluations on LLaMA-2 and LLaMA-3 show that DuoGPT outperforms state-of-the-art structured pruning methods by up to 9.17% accuracy at an iso-speedup of 1.39× compared to the baseline dense model. Code is available at Github.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

42 of 49 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
30 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  13. 2021Accelerating Sparse Deep Neural Networkscited by 223
  14. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  15. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  16. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  17. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  18. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
  19. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  20. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  21. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  22. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 44
  23. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
  24. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  25. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 21
  26. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  27. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9
  28. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  29. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  30. 2016Single Pass PCA of Matrix Productscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMscited by 1
  2. 2026Unified Static-Dynamic Pruning for Efficient LLM Inferencecited by 0
  3. 2026At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inferencecited by 0
  4. 2026Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inferencecited by 0
  5. 2026LaMoC: Loss-Aware Modular Compression for LLMscited by 0
  6. 2026Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decodingcited by 0