# Neural networks and rational functions

arXiv: https://arxiv.org/abs/1706.03301 · HTML: https://mlsanity.com/paper/1706.03301/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2017.

## Built on by (most cited first, 30 of 40)

- [Convolutional Neural Operators for robust and accurate learning of PDEs](https://mlsanity.com/paper/2302.01178/) (2023)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Kolmogorov-Arnold Transformer](https://mlsanity.com/paper/2409.10594/) (2024)
- [Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem](https://mlsanity.com/paper/2101.08286/) (2021)
- [rKAN: Rational Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2406.14495/) (2024)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power](https://mlsanity.com/paper/2205.13863/) (2022)
- [Neural tensor contractions and the expressive power of deep neural quantum states](https://mlsanity.com/paper/2103.10293/) (2021)
- [Parametric Complexity Bounds for Approximating PDEs with Neural Networks](https://mlsanity.com/paper/2103.02138/) (2021)
- [Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks](https://mlsanity.com/paper/2107.10234/) (2021)
- [Learning to Optimize for Reinforcement Learning](https://mlsanity.com/paper/2302.01470/) (2023)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Covering Number of Real Algebraic Varieties and Beyond: Improved Bounds and Applications](https://mlsanity.com/paper/2311.05116/) (2023)
- [Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks](https://mlsanity.com/paper/2404.03761/) (2024)
- [Statistical guarantees for denoising reflected diffusion models](https://mlsanity.com/paper/2411.01563/) (2024)
- [On Regularizing Coordinate-MLPs](https://mlsanity.com/paper/2202.00790/) (2022)
- [Transformers with Learnable Activation Functions](https://mlsanity.com/paper/2208.14111/) (2022)
- [Cauchy activation function and XNet](https://mlsanity.com/paper/2409.19221/) (2024)
- [SignReLU neural network and its approximation ability](https://mlsanity.com/paper/2210.10264/) (2022)
- [Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning](https://mlsanity.com/paper/2509.13240/) (2025)
- [Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings](https://mlsanity.com/paper/2505.15353/) (2025)
- [Graph Neural Networks Need Cluster-Normalize-Activate Modules](https://mlsanity.com/paper/2412.04064/) (2024)
- [C1-approximation with rational functions and rational neural networks](https://mlsanity.com/paper/2508.19672/) (2025)
- [Rational ANOVA Networks](https://mlsanity.com/paper/2602.04006/) (2026)
- [Rational Neural Networks have Expressivity Advantages](https://mlsanity.com/paper/2602.12390/) (2026)
- [STENCIL-NET: Data-driven solution-adaptive discretization of partial differential equations](https://mlsanity.com/paper/2101.06182/) (2021)
- [PO-CKAN:Physics Informed Deep Operator Kolmogorov Arnold Networks with Chunk Rational Structure](https://mlsanity.com/paper/2510.08795/) (2025)
- [Quantitative Approximation Rates for Group Equivariant Learning](https://mlsanity.com/paper/2602.20370/) (2026)

## Abstract (arXiv)

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree O(polylog(1/ε)) which is ε-close, and similarly for any rational function there exists a ReLU network of size O(polylog(1/ε)) which is ε-close. By contrast, polynomials need degree Omega(poly(1/ε)) to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.