# Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2108.12943 · HTML: https://mlsanity.com/paper/2108.12943/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 10 of 10)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Mish: A Self Regularized Non-Monotonic Activation Function](https://mlsanity.com/paper/1908.08681/) (2019)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Neural Network with Unbounded Activation Functions is Universal Approximator](https://mlsanity.com/paper/1505.03654/) (2015)
- [Why ReLU Units Sometimes Die: Analysis of Single-Unit Error Backpropagation in Neural Networks](https://mlsanity.com/paper/1812.05981/) (2018)
- [Improving Deep Neural Network with Multiple Parametric Exponential Linear Units](https://mlsanity.com/paper/1606.00305/) (2016)
- [A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks](https://mlsanity.com/paper/1811.03980/) (2018)

## Built on by (most cited first, 13 of 13)

- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [A survey on recently proposed activation functions for Deep Learning](https://mlsanity.com/paper/2204.02921/) (2022)
- [Embeddings between Barron spaces with higher order activation functions](https://mlsanity.com/paper/2305.15839/) (2023)
- [Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently](https://mlsanity.com/paper/2304.09759/) (2023)
- [Modeling non-linear Effects with Neural Networks in Relational Event Models](https://mlsanity.com/paper/2312.12357/) (2023)
- [PReLU: Yet Another Single-Layer Solution to the XOR Problem](https://mlsanity.com/paper/2409.10821/) (2024)
- [ASU-CNN: An Efficient Deep Architecture for Image Classification and Feature Visualizations](https://mlsanity.com/paper/2305.19146/) (2023)
- [Spontaneous Kolmogorov-Arnold Geometry in Shallow MLPs](https://mlsanity.com/paper/2509.12326/) (2025)
- [Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity](https://mlsanity.com/paper/2610.00149/) (2026)
- [HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation](https://mlsanity.com/paper/2110.15440/) (2021)
- [Evaluating CNN with Oscillatory Activation Function](https://mlsanity.com/paper/2211.06878/) (2022)
- [Efficient Vectorized Backpropagation Algorithms for Training Feedforward Networks Composed of Quadratic Neurons](https://mlsanity.com/paper/2310.02901/) (2023)
- [Multi-Behavioral Evolved Substrates Through Neuromodulation and Activation Selection](https://mlsanity.com/paper/2610.00148/) (2026)

## Abstract (arXiv)

Convolutional neural networks have been successful in solving many socially important and economically significant problems. This ability to learn complex high-dimensional functions hierarchically can be attributed to the use of nonlinear activation functions. A key discovery that made training deep networks feasible was the adoption of the Rectified Linear Unit (ReLU) activation function to alleviate the vanishing gradient problem caused by using saturating activation functions. Since then, many improved variants of the ReLU activation have been proposed. However, a majority of activation functions used today are non-oscillatory and monotonically increasing due to their biological plausibility. This paper demonstrates that oscillatory activation functions can improve gradient flow and reduce network size. Two theorems on limits of non-oscillatory activation functions are presented. A new oscillatory activation function called Growing Cosine Unit(GCU) defined as C(z) = zcos z that outperforms Sigmoids, Swish, Mish and ReLU on a variety of architectures and benchmarks is presented. The GCU activation has multiple zeros enabling single GCU neurons to have multiple hyperplanes in the decision boundary. This allows single GCU neurons to learn the XOR function without feature engineering. Experimental results indicate that replacing the activation function in the convolution layers with the GCU activation function significantly improves performance on CIFAR-10, CIFAR-100 and Imagenette.