# Network Deconvolution

arXiv: https://arxiv.org/abs/1905.11926 · HTML: https://mlsanity.com/paper/1905.11926/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 9 of 9)

- [Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training](https://mlsanity.com/paper/2103.00673/) (2021)
- [Modulate Your Spectrum in Self-Supervised Learning](https://mlsanity.com/paper/2305.16789/) (2023)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [Improving Generalization of Batch Whitening by Convolutional Unit Optimization](https://mlsanity.com/paper/2108.10629/) (2021)
- [Exploiting Invariance in Training Deep Neural Networks](https://mlsanity.com/paper/2103.16634/) (2021)
- [Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation](https://mlsanity.com/paper/2507.20284/) (2025)
- [Universal Learned Image Compression With Low Computational Cost](https://mlsanity.com/paper/2206.11599/) (2022)
- [On the limits of neural network explainability via descrambling](https://mlsanity.com/paper/2301.07820/) (2023)
- [[Re] Network Deconvolution](https://mlsanity.com/paper/2410.01189/) (2024)

## Abstract (arXiv)

Convolution is a central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image. However, because of the strong correlations in real-world image data, convolutional kernels are in effect re-learning redundant data. In this work, we show that this redundancy has made neural network training challenging, and propose network deconvolution, a procedure which optimally removes pixel-wise and channel-wise correlations before the data is fed into each layer. Network deconvolution can be efficiently calculated at a fraction of the computational cost of a convolution layer. We also show that the deconvolution filters in the first layer of the network resemble the center-surround structure found in biological neurons in the visual regions of the brain. Filtering with such kernels results in a sparse representation, a desired property that has been missing in the training of neural networks. Learning from the sparse representation promotes faster convergence and superior results without the use of batch normalization. We apply our network deconvolution operation to 10 modern neural network models by replacing batch normalization within each. Extensive experiments show that the network deconvolution operation is able to deliver performance improvement in all cases on the CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Cityscapes, and ImageNet datasets.