# Compressing Deep Neural Networks via Layer Fusion

arXiv: https://arxiv.org/abs/2007.14917 · HTML: https://mlsanity.com/paper/2007.14917/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [Layer Folding: Neural Network Depth Reduction using Activation Linearization](https://mlsanity.com/paper/2106.09309/) (2021)
- [PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation](https://mlsanity.com/paper/2103.00053/) (2021)
- [Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks](https://mlsanity.com/paper/2508.00043/) (2025)
- [Performance and Complexity Trade-off Optimization of Speech Models During Training](https://mlsanity.com/paper/2601.13704/) (2026)

## Abstract (arXiv)

This paper proposes textitlayer fusion - a model compression technique that discovers which weights to combine and then fuses weights of similar fully-connected, convolutional and attention layers. Layer fusion can significantly reduce the number of layers of the original network with little additional computation overhead, while maintaining competitive performance. From experiments on CIFAR-10, we find that various deep convolution neural networks can remain within 2% accuracy points of the original networks up to a compression ratio of 3.33 when iteratively retrained with layer fusion. For experiments on the WikiText-2 language modelling dataset where pretrained transformer models are used, we achieve compression that leads to a network that is 20% of its original size while being within 5 perplexity points of the original network. We also find that other well-established compression techniques can achieve competitive performance when compared to their original networks given a sufficient number of retraining steps. Generally, we observe a clear inflection point in performance as the amount of compression increases, suggesting a bound on the amount of compression that can be achieved before an exponential degradation in performance.