# Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity

arXiv: https://arxiv.org/abs/2211.05624 · HTML: https://mlsanity.com/paper/2211.05624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [Extrapolation and learning equations](https://mlsanity.com/paper/1610.02995/) (2016)
- [Neural Arithmetic Logic Units](https://mlsanity.com/paper/1808.00508/) (2018)
- [Identity Crisis: Memorization and Generalization under Extreme Overparameterization](https://mlsanity.com/paper/1902.04698/) (2019)
- [Neural Arithmetic Units](https://mlsanity.com/paper/2001.05016/) (2020)
- [Neural Power Units](https://mlsanity.com/paper/2006.01681/) (2020)
- [A Primer for Neural Arithmetic Logic Modules](https://mlsanity.com/paper/2101.09530/) (2021)
- [Measuring Arithmetic Extrapolation Performance](https://mlsanity.com/paper/1910.01888/) (2019)
- [iNALU: Improved Neural Arithmetic Logic Unit](https://mlsanity.com/paper/2003.07629/) (2020)
- [Analogical Reasoning for Visually Grounded Language Acquisition](https://mlsanity.com/paper/2007.11668/) (2020)

## Abstract (arXiv)

Multilayer Perceptrons struggle to learn certain simple arithmetic tasks. Specialist neural modules for arithmetic can outperform classical architectures with gains in extrapolation, interpretability and convergence speeds, but are highly sensitive to the training range. In this paper, we show that Neural Multiplication Units (NMUs) are unable to reliably learn tasks as simple as multiplying two inputs when given different training ranges. Causes of failure are linked to inductive and input biases which encourage convergence to solutions in undesirable optima. A solution, the stochastic NMU (sNMU), is proposed to apply reversible stochasticity, encouraging avoidance of such optima whilst converging to the true solution. Empirically, we show that stochasticity provides improved robustness with the potential to improve learned representations of upstream networks for numerical and image tasks.