# Filter Distillation for Network Compression

arXiv: https://arxiv.org/abs/1807.10585 · HTML: https://mlsanity.com/paper/1807.10585/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2018.

## Built on by (most cited first, 17 of 17)

- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)
- [Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining](https://mlsanity.com/paper/2407.19126/) (2024)
- [Toward Compact Deep Neural Networks via Energy-Aware Pruning](https://mlsanity.com/paper/2103.10858/) (2021)
- [Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruning](https://mlsanity.com/paper/2209.13590/) (2022)
- [Revealing the Utilized Rank of Subspaces of Learning in Neural Networks](https://mlsanity.com/paper/2407.04797/) (2024)
- [Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments](https://mlsanity.com/paper/2408.03274/) (2024)
- [Receding Neuron Importances for Structured Pruning](https://mlsanity.com/paper/2204.06404/) (2022)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)
- [i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery](https://mlsanity.com/paper/2112.04905/) (2021)
- [Supervised Robustness-preserving Data-free Neural Network Pruning](https://mlsanity.com/paper/2204.00783/) (2022)
- [Designing the Architecture of a Convolutional Neural Network Automatically for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2110.03877/) (2021)
- [Neural Network Compression via Effective Filter Analysis and Hierarchical Pruning](https://mlsanity.com/paper/2206.03596/) (2022)
- [Neural Language Model Pruning for Automatic Speech Recognition](https://mlsanity.com/paper/2310.03424/) (2023)
- [Robustness to distribution shifts of compressed networks for edge devices](https://mlsanity.com/paper/2401.12014/) (2024)
- [Subspace Node Pruning](https://mlsanity.com/paper/2405.17506/) (2024)

## Abstract (arXiv)

In this paper we introduce Principal Filter Analysis (PFA), an easy to use and effective method for neural network compression. PFA exploits the correlation between filter responses within network layers to recommend a smaller network that maintain as much as possible the accuracy of the full model. We propose two algorithms: the first allows users to target compression to specific network property, such as number of trainable variable (footprint), and produces a compressed model that satisfies the requested property while preserving the maximum amount of spectral energy in the responses of each layer, while the second is a parameter-free heuristic that selects the compression used at each layer by trying to mimic an ideal set of uncorrelated responses. Since PFA compresses networks based on the correlation of their responses we show in our experiments that it gains the additional flexibility of adapting each architecture to a specific domain while compressing. PFA is evaluated against several architectures and datasets, and shows considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. Our tests show that PFA is competitive with state-of-the-art approaches while removing adoption barriers thanks to its practical implementation, intuitive philosophy and ease of use.