# Smooth activations and reproducibility in deep networks

arXiv: https://arxiv.org/abs/2010.09931 · HTML: https://mlsanity.com/paper/2010.09931/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Deterministic Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2302.08300/) (2023)
- [Reproducibility in Optimization: Theoretical Framework and Limits](https://mlsanity.com/paper/2202.04598/) (2022)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions](https://mlsanity.com/paper/2409.09778/) (2024)
- [Synthesizing Irreproducibility in Deep Networks](https://mlsanity.com/paper/2102.10696/) (2021)
- [Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environments](https://mlsanity.com/paper/2204.04735/) (2022)
- [Disentangling Model Multiplicity in Deep Learning](https://mlsanity.com/paper/2206.08890/) (2022)
- [Instability in clinical risk stratification models using deep learning](https://mlsanity.com/paper/2211.10828/) (2022)
- [On The Fairness Impacts of Hardware Selection in Machine Learning](https://mlsanity.com/paper/2312.03886/) (2023)
- [Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength](https://mlsanity.com/paper/2110.06435/) (2021)
- [Selective Matching Losses -- Not All Scores Are Created Equal](https://mlsanity.com/paper/2506.04446/) (2025)
- [dynActivation: A Trainable Activation Family for Adaptive Nonlinearity](https://mlsanity.com/paper/2603.22154/) (2026)
- [Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective](https://mlsanity.com/paper/2610.14394/) (2026)

## Abstract (arXiv)

Deep networks are gradually penetrating almost every domain in our lives due to their amazing success. However, with substantive performance accuracy improvements comes the price of irreproducibility. Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems. The popular Rectified Linear Unit (ReLU) activation has been key to recent success of deep networks. We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks. We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs. We propose a new family of activations; Smooth ReLU (SmeLU), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap. SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility. We generalize SmeLU to give even more flexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctified Smooth Continuous Unit (RESCU) activations. Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular.