# EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

arXiv: https://arxiv.org/abs/1905.05934 · HTML: https://mlsanity.com/paper/1905.05934/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2019.

## Built on by (most cited first, 30 of 98)

- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://mlsanity.com/paper/2107.03356/) (2021)
- [Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs](https://mlsanity.com/paper/2407.00945/) (2024)
- [How Well Do Sparse Imagenet Models Transfer?](https://mlsanity.com/paper/2111.13445/) (2021)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Hessian-Aware Pruning and Optimal Neural Implant](https://mlsanity.com/paper/2101.08940/) (2021)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [The LLM Surgeon](https://mlsanity.com/paper/2312.17244/) (2023)
- [Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients](https://mlsanity.com/paper/2202.08132/) (2022)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://mlsanity.com/paper/2505.07004/) (2025)
- [Pruning's Effect on Generalization Through the Lens of Training and Regularization](https://mlsanity.com/paper/2210.13738/) (2022)
- [SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning](https://mlsanity.com/paper/2110.11395/) (2021)
- [Dual Lottery Ticket Hypothesis](https://mlsanity.com/paper/2203.04248/) (2022)
- [Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!](https://mlsanity.com/paper/2303.02141/) (2023)
- [Federated Domain Generalization: A Survey](https://mlsanity.com/paper/2306.01334/) (2023)
- [Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network](https://mlsanity.com/paper/2204.02738/) (2022)
- [Winning the Lottery Ahead of Time: Efficient Early Network Pruning](https://mlsanity.com/paper/2206.10451/) (2022)
- [CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models](https://mlsanity.com/paper/2210.09223/) (2022)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)
- [Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning](https://mlsanity.com/paper/2301.05219/) (2023)
- [Using Degeneracy in the Loss Landscape for Mechanistic Interpretability](https://mlsanity.com/paper/2405.10927/) (2024)

## Abstract (arXiv)

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structured pruning methods in this basis. As opposed to existing Hessian-based pruning algorithms which do pruning in parameter coordinates, our method works in the KFE where different weights are approximately independent, enabling accurate pruning and fast computation. We demonstrate empirically the effectiveness of the proposed method through extensive experiments. In particular, we highlight that the improvements are especially significant for more challenging datasets and networks. With negligible loss of accuracy, an iterative-pruning version gives a 10× reduction in model size and a 8× reduction in FLOPs on wide ResNet32.