MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning
Not on the map. Why?
Abstract (arXiv)
Determining the optimal size of a neural network is critical, as it directly impacts runtime performance and memory usage. Pruning is a well-established model compression technique that reduces the size of neural networks while mathematically guaranteeing accuracy preservation. However, many recent pruning methods overlook the global contributions of individual model components, making it difficult to ensure that a pruned model meets the desired dataset and performance requirements. To address these challenges, we developed a new pruning algorithm, MPruner, that leverages mutual information through vector similarity. MPruner utilizes layer clustering with the Centered Kernel Alignment (CKA) similarity metric, allowing us to incorporate global information from the neural network for more precise and efficient layer-wise pruning. We evaluated MPruner across various architectures and configurations, demonstrating its versatility and providing practical guidelines. MPruner achieved up to a 50% reduction in parameters and memory usage for CNN and transformer-based models, with minimal to no loss in accuracy.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 24 Aug 2024.
Builds on
29 of 41 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
17 more
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
- 2019Revealing the Dark Secrets of BERTcited by 300
- 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
- 2021Block Pruning For Faster Transformerscited by 176
- 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
- 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
- 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
- 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
- 2023Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --cited by 71
- 2016Pruning Convolutional Neural Networks for Resource Efficient Inferencecited by 47
- 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
- 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
- 2014A Kernel Independence Test for Random Processescited by 18
- 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
- 2020MINT: Deep Network Compression via Mutual Information-based Neuron Trimmingcited by 11
- 2023Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognitioncited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.
- 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
- 2025Dynamic Low-Rank Sparse Adaptation for Large Language Models20 Feb 2025