# Neuron Merging: Compensating for Pruned Neurons

arXiv: https://arxiv.org/abs/2010.13160 · HTML: https://mlsanity.com/paper/2010.13160/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 24 of 24)

- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Unified Data-Free Compression: Pruning and Quantization without Fine-Tuning](https://mlsanity.com/paper/2308.07209/) (2023)
- [RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks](https://mlsanity.com/paper/2105.14797/) (2021)
- [RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging](https://mlsanity.com/paper/2110.01397/) (2021)
- [Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models](https://mlsanity.com/paper/2308.03449/) (2023)
- [Data-Free Quantization via Mixed-Precision Compensation without Fine-Tuning](https://mlsanity.com/paper/2307.00498/) (2023)
- [Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy](https://mlsanity.com/paper/2111.09635/) (2021)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [TropNNC: Structured Neural Network Compression Using Tropical Geometry](https://mlsanity.com/paper/2409.03945/) (2024)
- [Image Complexity Guided Network Compression for Biomedical Image Segmentation](https://mlsanity.com/paper/2107.02927/) (2021)
- [STAT: Shrinking Transformers After Training](https://mlsanity.com/paper/2406.00061/) (2024)
- [Training-Free Restoration of Pruned Neural Networks](https://mlsanity.com/paper/2502.08474/) (2025)
- [PRANC: Pseudo RAndom Networks for Compacting deep models](https://mlsanity.com/paper/2206.08464/) (2022)
- [AutoSculpt: A Pattern-based Model Auto-pruning Framework Using Reinforcement Learning and Graph Learning](https://mlsanity.com/paper/2412.18091/) (2024)
- [Sharp asymptotics on the compression of two-layer neural networks](https://mlsanity.com/paper/2205.08199/) (2022)
- [Hyperparameter Optimization with Neural Network Pruning](https://mlsanity.com/paper/2205.08695/) (2022)
- [When do Convolutional Neural Networks Stop Learning?](https://mlsanity.com/paper/2403.02473/) (2024)
- [Subspace Node Pruning](https://mlsanity.com/paper/2405.17506/) (2024)
- [TT-MPD: Test Time Model Pruning and Distillation](https://mlsanity.com/paper/2412.07114/) (2024)
- [NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devices](https://mlsanity.com/paper/2504.03118/) (2025)
- [Squeeze-Release: Iterative Pruning with Exact Structural Minimization](https://mlsanity.com/paper/2606.14346/) (2026)

## Abstract (arXiv)

Network pruning is widely used to lighten and accelerate neural network models. Structured network pruning discards the whole neuron or filter, leading to accuracy loss. In this work, we propose a novel concept of neuron merging applicable to both fully connected layers and convolution layers, which compensates for the information loss due to the pruned neurons/filters. Neuron merging starts with decomposing the original weights into two matrices/tensors. One of them becomes the new weights for the current layer, and the other is what we name a scaling matrix, guiding the combination of neurons. If the activation function is ReLU, the scaling matrix can be absorbed into the next layer under certain conditions, compensating for the removed neurons. We also propose a data-free and inexpensive method to decompose the weights by utilizing the cosine similarity between neurons. Compared to the pruned model with the same topology, our merged model better preserves the output feature map of the original model; thus, it maintains the accuracy after pruning without fine-tuning. We demonstrate the effectiveness of our approach over network pruning for various model architectures and datasets. As an example, for VGG-16 on CIFAR-10, we achieve an accuracy of 93.16% while reducing 64% of total parameters, without any fine-tuning. The code can be found here: https://github.com/friendshipkim/neuron-merging