# Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients

arXiv: https://arxiv.org/abs/2605.01702 · HTML: https://mlsanity.com/paper/2605.01702/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 14 of 14)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [FP8 Formats for Deep Learning](https://mlsanity.com/paper/2209.05433/) (2022)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Expressive Power of ReLU and Step Networks under Floating-Point Operations](https://mlsanity.com/paper/2401.15121/) (2024)
- [Floating-Point Neural Networks Are Provably Robust Universal Approximators](https://mlsanity.com/paper/2506.16065/) (2025)
- [On the Expressive Power of Floating-Point Transformers](https://mlsanity.com/paper/2601.16450/) (2026)

## Abstract (arXiv)

Theoretical studies show that for any differentiable function on a compact domain, there exists a neural network that approximates both the function values and gradients. However, such a result cannot be used in practice since it assumes real parameters and exact internal operations. In contrast, real implementations only use a finite subset of reals and machine operations with round-off errors. In this work, we investigate whether a similar result holds for neural networks under floating-point arithmetic, when the gradient with respect to the input is computed by the automatic differentiation algorithm D^mathttAD. We first show that given a floating-point function φ (e.g., a loss function), arbitrary function values and gradients can be represented by a floating-point network f and D^mathttAD(φcirc f), respectively. We further extend this result: given φ1,dots,φn, D^mathttAD(φicirc f) can simultaneously represent arbitrary gradients while f represents the target values, under mild conditions. Our results hold for practical activation functions, e.g., ReLU, ELU, GeLU, Swish, Sigmoid, and tanh.