# SPLASH: Learnable Activation Functions for Improving Accuracy and Adversarial Robustness

arXiv: https://arxiv.org/abs/2006.08947 · HTML: https://mlsanity.com/paper/2006.08947/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 18 of 18)

- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Parameterizing Activation Functions for Adversarial Robustness](https://mlsanity.com/paper/2110.05626/) (2021)
- [MeanSparse: Post-Training Robustness Enhancement Through Mean-Centered Feature Sparsification](https://mlsanity.com/paper/2406.05927/) (2024)
- [The Quarks of Attention](https://mlsanity.com/paper/2202.08371/) (2022)
- [Quantum Mechanics and Machine Learning Synergies: Graph Attention Neural Networks to Predict Chemical Reactivity](https://mlsanity.com/paper/2103.14536/) (2021)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Optimal Activation Functions for the Random Features Regression Model](https://mlsanity.com/paper/2206.01332/) (2022)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [Efficient Search for Customized Activation Functions with Gradient Descent](https://mlsanity.com/paper/2408.06820/) (2024)
- [Kolmogorov-Arnold Network for Transistor Compact Modeling](https://mlsanity.com/paper/2503.15209/) (2025)
- [On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions](https://mlsanity.com/paper/2510.15327/) (2025)
- [Deep Learning Models of the Discrete Component of the Galactic Interstellar Gamma-Ray Emission](https://mlsanity.com/paper/2206.02819/) (2022)
- [A Theory of Synaptic Neural Balance: From Local to Global Order](https://mlsanity.com/paper/2405.09688/) (2024)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [Polynomial, trigonometric, and tropical activations](https://mlsanity.com/paper/2502.01247/) (2025)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)
- [FlexAct: Why Learn when you can Pick?](https://mlsanity.com/paper/2601.06441/) (2026)

## Abstract (arXiv)

We introduce SPLASH units, a class of learnable activation functions shown to simultaneously improve the accuracy of deep neural networks while also improving their robustness to adversarial attacks. SPLASH units have both a simple parameterization and maintain the ability to approximate a wide range of non-linear functions. SPLASH units are: 1) continuous; 2) grounded (f(0) = 0); 3) use symmetric hinges; and 4) the locations of the hinges are derived directly from the data (i.e. no learning required). Compared to nine other learned and fixed activation functions, including ReLU and its variants, SPLASH units show superior performance across three datasets (MNIST, CIFAR-10, and CIFAR-100) and four architectures (LeNet5, All-CNN, ResNet-20, and Network-in-Network). Furthermore, we show that SPLASH units significantly increase the robustness of deep neural networks to adversarial attacks. Our experiments on both black-box and open-box adversarial attacks show that commonly-used architectures, namely LeNet5, All-CNN, ResNet-20, and Network-in-Network, can be up to 31% more robust to adversarial attacks by simply using SPLASH units instead of ReLUs.