# DC is all you need: describing ReLU from a signal processing standpoint

arXiv: https://arxiv.org/abs/2407.16556 · HTML: https://mlsanity.com/paper/2407.16556/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 8 of 8)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes](https://mlsanity.com/paper/1706.10239/) (2017)
- [Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective](https://mlsanity.com/paper/2110.03095/) (2021)
- [Image classification in frequency domain with 2SReLU: a second harmonics superposition activation function](https://mlsanity.com/paper/2006.10853/) (2020)
- [KID-PPG: Knowledge Informed Deep Learning for Extracting Heart Rate from a Smartwatch](https://mlsanity.com/paper/2405.09559/) (2024)
- [Probabilistic Verification of ReLU Neural Networks via Characteristic Functions](https://mlsanity.com/paper/2212.01544/) (2022)

## Built on by (most cited first, 3 of 3)

- [Spiking Neural Networks Need High Frequency Information](https://mlsanity.com/paper/2505.18608/) (2025)
- [Time series saliency maps: explaining models across multiple domains](https://mlsanity.com/paper/2505.13100/) (2025)
- [Don't Think It Twice: Exploit Shift Invariance for Efficient Online Streaming Inference of CNNs](https://mlsanity.com/paper/2408.03223/) (2024)

## Abstract (arXiv)

Non-linear activation functions are crucial in Convolutional Neural Networks. However, until now they have not been well described in the frequency domain. In this work, we study the spectral behavior of ReLU, a popular activation function. We use the ReLU's Taylor expansion to derive its frequency domain behavior. We demonstrate that ReLU introduces higher frequency oscillations in the signal and a constant DC component. Furthermore, we investigate the importance of this DC component, where we demonstrate that it helps the model extract meaningful features related to the input frequency content. We accompany our theoretical derivations with experiments and real-world examples. First, we numerically validate our frequency response model. Then we observe ReLU's spectral behavior on two example models and a real-world one. Finally, we experimentally investigate the role of the DC component introduced by ReLU in the CNN's representations. Our results indicate that the DC helps to converge to a weight configuration that is close to the initial random weights.