# BrowNNe: Brownian Nonlocal Neurons & Activation Functions

arXiv: https://arxiv.org/abs/2406.15617 · HTML: https://mlsanity.com/paper/2406.15617/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [On the Convergence of SGD with Biased Gradients](https://mlsanity.com/paper/2008.00051/) (2020)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [ProbAct: A Probabilistic Activation Function for Deep Neural Networks](https://mlsanity.com/paper/1905.10761/) (2019)
- [On Biased Stochastic Gradient Estimation](https://mlsanity.com/paper/1906.01133/) (2019)
- [Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling](https://mlsanity.com/paper/1806.00681/) (2018)
- [Function approximation by deep networks](https://mlsanity.com/paper/1905.12882/) (2019)
- [Optimization and Learning With Nonlocal Calculus](https://mlsanity.com/paper/2012.07013/) (2020)
- [Nonlocal gradients in bounded domains motivated by Continuum Mechanics: Fundamental Theorem of Calculus and embeddings](https://mlsanity.com/paper/2201.08793/) (2022)

## Abstract (arXiv)

It is generally thought that the use of stochastic activation functions in deep learning architectures yield models with superior generalization abilities. However, a sufficiently rigorous statement and theoretical proof of this heuristic is lacking in the literature. In this paper, we provide several novel contributions to the literature in this regard. Defining a new notion of nonlocal directional derivative, we analyze its theoretical properties (existence and convergence). Second, using a probabilistic reformulation, we show that nonlocal derivatives are epsilon-sub gradients, and derive sample complexity results for convergence of stochastic gradient descent-like methods using nonlocal derivatives. Finally, using our analysis of the nonlocal gradient of Holder continuous functions, we observe that sample paths of Brownian motion admit nonlocal directional derivatives, and the nonlocal derivatives of Brownian motion are seen to be Gaussian processes with computable mean and standard deviation. Using the theory of nonlocal directional derivatives, we solve a highly nondifferentiable and nonconvex model problem of parameter estimation on image articulation manifolds. Using Brownian motion infused ReLU activation functions with the nonlocal gradient in place of the usual gradient during backpropagation, we also perform experiments on multiple well-studied deep learning architectures. Our experiments indicate the superior generalization capabilities of Brownian neural activation functions in low-training data regimes, where the use of stochastic neurons beats the deterministic ReLU counterpart.