# Neural Arithmetic Units

arXiv: https://arxiv.org/abs/2001.05016 · HTML: https://mlsanity.com/paper/2001.05016/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2020.

## Built on by (most cited first, 28 of 28)

- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023)
- [MC-LSTM: Mass-Conserving LSTM](https://mlsanity.com/paper/2101.05186/) (2021)
- [Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model](https://mlsanity.com/paper/2109.04672/) (2021)
- [Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation](https://mlsanity.com/paper/2207.02536/) (2022)
- [Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers](https://mlsanity.com/paper/2105.07957/) (2021)
- [Asynchronous Neural Networks for Learning in Graphs](https://mlsanity.com/paper/2205.12245/) (2022)
- [Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks](https://mlsanity.com/paper/2502.16763/) (2025)
- [Logical Tasks for Measuring Extrapolation and Rule Comprehension](https://mlsanity.com/paper/2211.07727/) (2022)
- [How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis](https://mlsanity.com/paper/2511.03825/) (2025)
- [TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks](https://mlsanity.com/paper/2606.02384/) (2026)
- [State-Space Constraints Can Improve the Generalisation of the Differentiable Neural Computer to Input Sequences With Unseen Length](https://mlsanity.com/paper/2110.09138/) (2021)
- [Open World Learning Graph Convolution for Latency Estimation in Routing Networks](https://mlsanity.com/paper/2207.14643/) (2022)
- [Learning to solve arithmetic problems with a virtual abacus](https://mlsanity.com/paper/2301.06870/) (2023)
- [Even GPT-5.2 Can't Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs](https://mlsanity.com/paper/2601.15714/) (2026)
- [Learning Division with Neural Arithmetic Logic Modules](https://mlsanity.com/paper/2110.05177/) (2021)
- [Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity](https://mlsanity.com/paper/2211.05624/) (2022)
- [A Synthetic Pseudo-Autoencoder Invites Examination of Tacit Assumptions in Neural Network Design](https://mlsanity.com/paper/2506.12076/) (2025)
- [The Domain Mixed Unit: A New Neural Arithmetic Layer](https://mlsanity.com/paper/2509.08180/) (2025)
- [Neural Isomorphic Fields: A Transformer-based Algebraic Numerical Embedding](https://mlsanity.com/paper/2601.12095/) (2026)
- [BRo-JEPA: Learning Modular Transformations in Latent Space](https://mlsanity.com/paper/2606.01372/) (2026)
- [TriPLU: Bypassing the Gate with Direct Trilinear Product FFNs in Tiny Language Models](https://mlsanity.com/paper/2608.20360/) (2026)
- [Geometric Mean Pooling for Equal-Weight Multiplicative Coarse-Graining](https://mlsanity.com/paper/2609.21876/) (2026)
- [The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning](https://mlsanity.com/paper/2609.32927/) (2026)
- [Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents](https://mlsanity.com/paper/2610.02525/) (2026)
- [Arithmetic Actor Heads and Training Stabilization for Out-of-Distribution Reinforcement Learning](https://mlsanity.com/paper/2610.05143/) (2026)

## Abstract (arXiv)

Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units, converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values.