mlsanityData to 7 Oct 2026

The LLM Surgeon

arXiv 2312.17244 ↗First posted 28 Dec 2023cs.LG cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factored curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dynamic allocation of structures that can be removed as well as updates of remaining weights that account for the removal. We provide a general framework for unstructured, semi-structured and structured pruning and improve upon weight updates to capture more correlations between weights, while remaining computationally efficient. Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 20 of its 29 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

20 of 29 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  12. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
8 more
  1. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  2. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  3. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  4. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  5. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  6. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  7. 2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
  8. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 15.

36 papers.

Sort:

By area:

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  3. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  4. 2024A deeper look at depth pruning of LLMscited by 23
  5. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  6. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  7. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  8. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  9. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 10
  10. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  11. 2025Olica: Efficient Structured Pruning of Large Language Models without Retrainingcited by 6
  12. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
24 more
  1. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  2. 2025laplax -- Laplace Approximations with JAXcited by 5
  3. 2025Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Modelscited by 4
  4. 2025Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generationcited by 4
  5. 2026Optimal Brain Decomposition for Accurate LLM Low-Rank Approximationcited by 4
  6. 2024STAT: Shrinking Transformers After Trainingcited by 2
  7. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 2
  8. 2026Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initializationcited by 2
  9. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  10. 2025Elastic ViTs from Pretrained Models without Retrainingcited by 1
  11. 2024OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruningcited by 0
  12. 2025Accelerating Attention with Basis Decompositioncited by 0
  13. 2026Compressing LLMs with MoP: Mixture of Prunerscited by 0
  14. 2026FlattenGPT: Depth Compression for Transformer with Layer Flatteningcited by 0
  15. 2026DOT-MoE: Differentiable Optimal Transport for MoEficationcited by 0
  16. 2026SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compressioncited by 0
  17. 2026Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compressioncited by 0
  18. 2026Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimizationcited by 0
  19. 2026PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruningcited by 0
  20. 2026XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compressioncited by 0
  21. 2026Output-aware Residual Stream Pruning for Large Language Modelscited by 0
  22. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0
  23. 2026XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantizationcited by 0
  24. 2026ReCal: Calibrating Structured Pruning for On-Policy Distillation Recoverycited by 0