# Normalization-Equivariant Neural Networks with Application to Image Denoising

arXiv: https://arxiv.org/abs/2306.05037 · HTML: https://mlsanity.com/paper/2306.05037/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Learning Steerable Filters for Rotation Equivariant CNNs](https://mlsanity.com/paper/1711.07289/) (2017)
- [Universal Invariant and Equivariant Graph Neural Networks](https://mlsanity.com/paper/1905.04943/) (2019)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Rotation equivariant vector field networks](https://mlsanity.com/paper/1612.09346/) (2016)
- [Robust and interpretable blind image denoising via bias-free convolutional neural networks](https://mlsanity.com/paper/1906.05478/) (2019)
- [Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)](https://mlsanity.com/paper/1604.02313/) (2016)
- [Rotation Equivariant Siamese Networks for Tracking](https://mlsanity.com/paper/2012.13078/) (2020)
- [ZZ-Net: A Universal Rotation Equivariant Architecture for 2D Point Clouds](https://mlsanity.com/paper/2111.15341/) (2021)
- [Fully Unsupervised Probabilistic Noise2Void](https://mlsanity.com/paper/1911.12291/) (2019)

## Built on by (most cited first, 11 of 11)

- [Learning normalized image densities via dual score matching](https://mlsanity.com/paper/2506.05310/) (2025)
- [Normalization-equivariant Diffusion Models: Learning Posterior Samplers From Noisy And Partial Measurements](https://mlsanity.com/paper/2510.11964/) (2025)
- [On normalization-equivariance properties of supervised and unsupervised denoising methods: a survey](https://mlsanity.com/paper/2402.15352/) (2024)
- [Equivariant Denoisers for Image Restoration](https://mlsanity.com/paper/2412.05343/) (2024)
- [Double Blind Imaging with Generative Modeling](https://mlsanity.com/paper/2503.21501/) (2025)
- [Normalize Filters! Classical Wisdom for Deep Vision](https://mlsanity.com/paper/2506.04401/) (2025)
- [A unified framework of non-local parametric methods for image denoising](https://mlsanity.com/paper/2402.13816/) (2024)
- [The Normalized Difference Layer: A Differentiable Spectral Index Formulation for Deep Learning](https://mlsanity.com/paper/2601.06777/) (2026)
- [Learning Affine-Equivariant Proximal Operators](https://mlsanity.com/paper/2604.15556/) (2026)
- [Normalization Equivariance for Arbitrary Backbones, with Application to Image Denoising](https://mlsanity.com/paper/2605.08193/) (2026)
- [Physen-Noise2Noise: Physics-Guided Self-Supervised Defocus Deblurring with Bias Correction under Low-Light Conditions](https://mlsanity.com/paper/2605.24590/) (2026)

## Abstract (arXiv)

In many information processing systems, it may be desirable to ensure that any change of the input, whether by shifting or scaling, results in a corresponding change in the system response. While deep neural networks are gradually replacing all traditional automatic processing methods, they surprisingly do not guarantee such normalization-equivariance (scale + shift) property, which can be detrimental in many applications. To address this issue, we propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, including the ReLU (rectified linear unit), which are applied element-wise to the pre-activated neurons, should be completely removed from neural networks and replaced by better conditioned alternatives. To this end, we introduce affine-constrained convolutions and channel-wise sort pooling layers as surrogates and show that these two architectural modifications do preserve normalization-equivariance without loss of performance. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.