# MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning

arXiv: https://arxiv.org/abs/2408.13482 · HTML: https://mlsanity.com/paper/2408.13482/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --](https://mlsanity.com/paper/2301.03362/) (2023)
- [Pruning Convolutional Neural Networks for Resource Efficient Inference](https://mlsanity.com/paper/1611.06440/) (2016)
- [How fine can fine-tuning be? Learning efficient language models](https://mlsanity.com/paper/2004.14129/) (2020)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging](https://mlsanity.com/paper/2406.16330/) (2024)
- [A Kernel Independence Test for Random Processes](https://mlsanity.com/paper/1402.4501/) (2014)
- [MINT: Deep Network Compression via Mutual Information-based Neuron Trimming](https://mlsanity.com/paper/2003.08472/) (2020)
- [Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognition](https://mlsanity.com/paper/2303.02693/) (2023)

## Built on by (most cited first, 1 of 1)

- [Dynamic Low-Rank Sparse Adaptation for Large Language Models](https://mlsanity.com/paper/2502.14816/) (2025)

## Abstract (arXiv)

Determining the optimal size of a neural network is critical, as it directly impacts runtime performance and memory usage. Pruning is a well-established model compression technique that reduces the size of neural networks while mathematically guaranteeing accuracy preservation. However, many recent pruning methods overlook the global contributions of individual model components, making it difficult to ensure that a pruned model meets the desired dataset and performance requirements. To address these challenges, we developed a new pruning algorithm, MPruner, that leverages mutual information through vector similarity. MPruner utilizes layer clustering with the Centered Kernel Alignment (CKA) similarity metric, allowing us to incorporate global information from the neural network for more precise and efficient layer-wise pruning. We evaluated MPruner across various architectures and configurations, demonstrating its versatility and providing practical guidelines. MPruner achieved up to a 50% reduction in parameters and memory usage for CNN and transformer-based models, with minimal to no loss in accuracy.