# The LLM Surgeon

arXiv: https://arxiv.org/abs/2312.17244 · HTML: https://mlsanity.com/paper/2312.17244/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Practical Gauss-Newton Optimisation for Deep Learning](https://mlsanity.com/paper/1706.03662/) (2017)
- [The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models](https://mlsanity.com/paper/2203.07259/) (2022)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures](https://mlsanity.com/paper/2311.00636/) (2023)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Learning Layer-wise Equivariances Automatically using Gradients](https://mlsanity.com/paper/2310.06131/) (2023)
- [Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition](https://mlsanity.com/paper/2201.10285/) (2022)

## Built on by (most cited first, 30 of 38)

- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models](https://mlsanity.com/paper/2406.02924/) (2024)
- [DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models](https://mlsanity.com/paper/2410.11988/) (2024)
- [A deeper look at depth pruning of LLMs](https://mlsanity.com/paper/2407.16286/) (2024)
- [ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization](https://mlsanity.com/paper/2505.02819/) (2025)
- [2SSP: A Two-Stage Framework for Structured Pruning of LLMs](https://mlsanity.com/paper/2501.17771/) (2025)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning](https://mlsanity.com/paper/2501.15316/) (2025)
- [EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models](https://mlsanity.com/paper/2502.06663/) (2025)
- [Position: Curvature Matrices Should Be Democratized via Linear Operators](https://mlsanity.com/paper/2501.19183/) (2025)
- [Olica: Efficient Structured Pruning of Large Language Models without Retraining](https://mlsanity.com/paper/2506.08436/) (2025)
- [Efficient LLMs with AMP: Attention Heads and MLP Pruning](https://mlsanity.com/paper/2504.21174/) (2025)
- [SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot](https://mlsanity.com/paper/2506.09613/) (2025)
- [laplax -- Laplace Approximations with JAX](https://mlsanity.com/paper/2507.17013/) (2025)
- [Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models](https://mlsanity.com/paper/2501.19090/) (2025)
- [Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation](https://mlsanity.com/paper/2502.08690/) (2025)
- [Elastic ViTs from Pretrained Models without Retraining](https://mlsanity.com/paper/2510.17700/) (2025)
- [Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation](https://mlsanity.com/paper/2604.00821/) (2026)
- [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](https://mlsanity.com/paper/2507.03117/) (2025)
- [STAT: Shrinking Transformers After Training](https://mlsanity.com/paper/2406.00061/) (2024)
- [FlattenGPT: Depth Compression for Transformer with Layer Flattening](https://mlsanity.com/paper/2602.08858/) (2026)
- [Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization](https://mlsanity.com/paper/2605.29327/) (2026)
- [Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs](https://mlsanity.com/paper/2608.08506/) (2026)
- [DOT-MoE: Differentiable Optimal Transport for MoEfication](https://mlsanity.com/paper/2606.01666/) (2026)
- [SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression](https://mlsanity.com/paper/2606.23568/) (2026)
- [OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning](https://mlsanity.com/paper/2405.05957/) (2024)
- [Accelerating Attention with Basis Decomposition](https://mlsanity.com/paper/2510.01718/) (2025)
- [Compressing LLMs with MoP: Mixture of Pruners](https://mlsanity.com/paper/2602.06127/) (2026)
- [Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression](https://mlsanity.com/paper/2607.18284/) (2026)
- [Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization](https://mlsanity.com/paper/2608.12953/) (2026)

## Abstract (arXiv)

State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factored curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dynamic allocation of structures that can be removed as well as updates of remaining weights that account for the removal. We provide a general framework for unstructured, semi-structured and structured pruning and improve upon weight updates to capture more correlations between weights, while remaining computationally efficient. Experimentally, our method can prune rows and columns from a range of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance, and achieve state-of-the-art results in unstructured and semi-structured pruning of large language models.