# Deep Residual Networks with Exponential Linear Unit

arXiv: https://arxiv.org/abs/1604.04112 · HTML: https://mlsanity.com/paper/1604.04112/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2016.

## Built on by (most cited first, 12 of 12)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Kolmogorov-Arnold Convolutions: Design Principles and Empirical Studies](https://mlsanity.com/paper/2407.01092/) (2024)
- [Learning from Randomly Initialized Neural Network Features](https://mlsanity.com/paper/2202.06438/) (2022)
- [UPANets: Learning from the Universal Pixel Attention Networks](https://mlsanity.com/paper/2103.08640/) (2021)
- [Signing the Supermask: Keep, Hide, Invert](https://mlsanity.com/paper/2201.13361/) (2022)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [BA^2M: A Batch Aware Attention Module for Image Classification](https://mlsanity.com/paper/2103.15099/) (2021)
- [FatNet: High Resolution Kernels for Classification Using Fully Convolutional Optical Neural Networks](https://mlsanity.com/paper/2210.16914/) (2022)
- [Attentive Contractive Flow with Lipschitz-constrained Self-Attention](https://mlsanity.com/paper/2109.12135/) (2021)
- [Bayesian optimization for sparse neural networks with trainable activation functions](https://mlsanity.com/paper/2304.04455/) (2023)
- [Thermal Face Image Classification using Deep Learning Techniques](https://mlsanity.com/paper/2311.02314/) (2023)

## Abstract (arXiv)

Very deep convolutional neural networks introduced new problems like vanishing gradient and degradation. The recent successful contributions towards solving these problems are Residual and Highway Networks. These networks introduce skip connections that allow the information (from the input or those learned in earlier layers) to flow more into the deeper layers. These very deep models have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose the use of exponential linear unit instead of the combination of ReLU and Batch Normalization in Residual Networks. We show that this not only speeds up learning in Residual Networks but also improves the accuracy as the depth increases. It improves the test error on almost all data sets, like CIFAR-10 and CIFAR-100