# Rational neural networks

arXiv: https://arxiv.org/abs/2004.01902 · HTML: https://mlsanity.com/paper/2004.01902/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2020.

## Built on by (most cited first, 30 of 75)

- [Convolutional Neural Operators for robust and accurate learning of PDEs](https://mlsanity.com/paper/2302.01178/) (2023)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [A Mathematical Guide to Operator Learning](https://mlsanity.com/paper/2312.14688/) (2023)
- [Kolmogorov-Arnold Transformer](https://mlsanity.com/paper/2409.10594/) (2024)
- [Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem](https://mlsanity.com/paper/2101.08286/) (2021)
- [Data-driven discovery of Green's functions with human-understandable deep learning](https://mlsanity.com/paper/2105.00266/) (2021)
- [rKAN: Rational Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2406.14495/) (2024)
- [Learning elliptic partial differential equations with randomized linear algebra](https://mlsanity.com/paper/2102.00491/) (2021)
- [Learning Green's functions associated with time-dependent partial differential equations](https://mlsanity.com/paper/2204.12789/) (2022)
- [PDE-READ: Human-readable Partial Differential Equation Discovery using Deep Learning](https://mlsanity.com/paper/2111.00998/) (2021)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective](https://mlsanity.com/paper/2503.13165/) (2025)
- [Geometry of Polynomial Neural Networks](https://mlsanity.com/paper/2402.00949/) (2024)
- [Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometry](https://mlsanity.com/paper/2501.18915/) (2025)
- [PDE-LEARN: Using Deep Learning to Discover Partial Differential Equations from Noisy, Limited Data](https://mlsanity.com/paper/2212.04971/) (2022)
- [Reproducing Activation Function for Deep Learning](https://mlsanity.com/paper/2101.04844/) (2021)
- [Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks](https://mlsanity.com/paper/2107.10234/) (2021)
- [Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks](https://mlsanity.com/paper/2310.04867/) (2023)
- [Discovery of partial differential equations from highly noisy and sparse data with physics-informed information criterion](https://mlsanity.com/paper/2208.03322/) (2022)
- [Weak-PDE-LEARN: A Weak Form Based Approach to Discovering PDEs From Noisy, Limited Data](https://mlsanity.com/paper/2309.04699/) (2023)
- [Activation degree thresholds and expressiveness of polynomial neural networks](https://mlsanity.com/paper/2408.04569/) (2024)
- [Adaptable Adapters](https://mlsanity.com/paper/2205.01549/) (2022)
- [Learning to Optimize for Reinforcement Learning](https://mlsanity.com/paper/2302.01470/) (2023)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [On Fast Simulation of Dynamical System with Neural Vector Enhanced Numerical Solver](https://mlsanity.com/paper/2208.03680/) (2022)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11846/) (2025)
- [Covering Number of Real Algebraic Varieties and Beyond: Improved Bounds and Applications](https://mlsanity.com/paper/2311.05116/) (2023)
- [Operator learning without the adjoint](https://mlsanity.com/paper/2401.17739/) (2024)
- [PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification](https://mlsanity.com/paper/2409.17834/) (2024)

## Abstract (arXiv)

We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments.