mlsanityData to 7 Oct 2026

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

arXiv 2401.15024 ↗First posted 26 Jan 2024cs.LG cs.CLcited by 378

Not on the map. Why?

Cited by 34 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

Key facts. Cited by 378 papers in the map (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2016Layer Normalizationcited by 5,579
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
23 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  9. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  12. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  13. 2019The State of Sparsity in Deep Neural Networkscited by 471
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  16. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  17. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  18. 2023A Survey on Model Compression for Large Language Modelscited by 263
  19. 2021Accelerating Sparse Deep Neural Networkscited by 213
  20. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  21. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  22. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  23. 2020Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 88; 2025: 151; 2026: 139.

378 papers.

Sort:

By area:

  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  3. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  4. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  6. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  9. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  10. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  11. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  12. 2024CLLMs: Consistency Large Language Modelscited by 71
38 more
  1. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  2. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  3. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  4. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
  5. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  6. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  7. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
  8. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  9. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  10. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 35
  11. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  12. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inferencecited by 32
  13. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  14. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  15. 2024Accuracy is Not All You Needcited by 27
  16. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
  17. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 26
  18. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  19. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  20. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  21. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  22. 2024Numerical Pruning for Efficient Autoregressive Modelscited by 24
  23. 2024Your Transformer is Secretly Linearcited by 23
  24. 2024A deeper look at depth pruning of LLMscited by 23
  25. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  26. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  27. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  28. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  29. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  30. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  31. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  32. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  33. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  34. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
  35. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  36. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  37. 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
  38. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15

50 most cited of 378 All 378 (JSON)