# Universal Approximation with Quadratic Deep Networks

arXiv: https://arxiv.org/abs/1808.00098 · HTML: https://mlsanity.com/paper/1808.00098/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2018.

## Built on by (most cited first, 29 of 29)

- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [Interpretable Polynomial Neural Ordinary Differential Equations](https://mlsanity.com/paper/2208.05072/) (2022)
- [StEik: Stabilizing the Optimization of Neural Signed Distance Functions and Finer Shape Representation](https://mlsanity.com/paper/2305.18414/) (2023)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [FC-KAN: Function Combinations in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2409.01763/) (2024)
- [Feasibility-based Fixed Point Networks](https://mlsanity.com/paper/2104.14090/) (2021)
- [Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods](https://mlsanity.com/paper/2410.05592/) (2024)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Attention-embedded Quadratic Network (Qttention) for Effective and Interpretable Bearing Fault Diagnosis](https://mlsanity.com/paper/2206.00390/) (2022)
- [One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks](https://mlsanity.com/paper/2303.06316/) (2023)
- [Bayesian polynomial neural networks and polynomial neural ordinary differential equations](https://mlsanity.com/paper/2308.10892/) (2023)
- [Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2204.01707/) (2022)
- [Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods](https://mlsanity.com/paper/2412.01181/) (2024)
- [Sound and Complete Verification of Polynomial Networks](https://mlsanity.com/paper/2209.07235/) (2022)
- [Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction](https://mlsanity.com/paper/2505.17999/) (2025)
- [Second-Order Component Analysis for Fault Detection](https://mlsanity.com/paper/2103.07303/) (2021)
- [Tailored neural networks for learning optimal value functions in MPC](https://mlsanity.com/paper/2112.03975/) (2021)
- [A Quadratic Actor Network for Model-Free Reinforcement Learning](https://mlsanity.com/paper/2103.06617/) (2021)
- [SQ-Swin: a Pretrained Siamese Quadratic Swin Transformer for Lettuce Browning Prediction](https://mlsanity.com/paper/2209.07683/) (2022)
- [Neural Networks Trained by Weight Permutation are Universal Approximators](https://mlsanity.com/paper/2407.01033/) (2024)
- [Plant Species Recognition with Optimized 3D Polynomial Neural Networks and Variably Overlapping Time-Coherent Sliding Window](https://mlsanity.com/paper/2203.02611/) (2022)
- [Continuous Function Structured in Multilayer Perceptron for Global Optimization](https://mlsanity.com/paper/2303.04623/) (2023)
- [Rational Neural Network Controllers](https://mlsanity.com/paper/2307.06287/) (2023)
- [Classification with neural networks with quadratic decision functions](https://mlsanity.com/paper/2401.10710/) (2024)
- [xMLP: Revolutionizing Private Inference with Exclusive Square Activation](https://mlsanity.com/paper/2403.08024/) (2024)
- [Enhancing Expressivity of Quantum Neural Networks Based on the SWAP test](https://mlsanity.com/paper/2506.16938/) (2025)
- [Structural Disentanglement in Bilinear MLPs via Architectural Inductive Bias](https://mlsanity.com/paper/2602.05635/) (2026)
- [No More DeLuLu: Physics-Inspired Kernel Networks for Geometrically-Grounded Neural Computation](https://mlsanity.com/paper/2603.12276/) (2026)
- [Medial Axis Aware Learning of Signed Distance Functions](https://mlsanity.com/paper/2604.16512/) (2026)

## Abstract (arXiv)

Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. Regarding this, we ask four basic questions in this paper: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? (4) To approximate the same class of functions with the same error bound, is a quantized quadratic network able to enjoy a lower number of weights than a quantized conventional network? Our main contributions are the four interconnected theorems shedding light upon these four questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, compact architecture and computational capacity respectively.