# One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks

arXiv: https://arxiv.org/abs/2303.06316 · HTML: https://mlsanity.com/paper/2303.06316/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 119 references that are in the map.
- First posted on arXiv 11 Mar 2023.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework](https://mlsanity.com/paper/2202.07123/) (2022)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation](https://mlsanity.com/paper/1802.06955/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review](https://mlsanity.com/paper/2004.05224/) (2020)
- [ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statistics](https://mlsanity.com/paper/1908.06295/) (2019)
- [Involution: Inverting the Inherence of Convolution for Visual Recognition](https://mlsanity.com/paper/2103.06255/) (2021)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [Deep Polynomial Neural Networks](https://mlsanity.com/paper/2006.13026/) (2020)
- [Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex](https://mlsanity.com/paper/1511.00083/) (2015)

## Built on by (most cited first, 6 of 6)

- [No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks](https://mlsanity.com/paper/2405.02369/) (2024)
- [Infinite-Dimensional Feature Interaction](https://mlsanity.com/paper/2405.13972/) (2024)
- [Cloud-RAIN: Point Cloud Analysis with Reflectional Invariance](https://mlsanity.com/paper/2305.07814/) (2023)
- [MixFunn: A Neural Network for Differential Equations with Improved Generalization and Interpretability](https://mlsanity.com/paper/2503.22528/) (2025)
- [QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks](https://mlsanity.com/paper/2510.03276/) (2025)
- [TNLearn: An Open Source Python Package for Task-based Neurons](https://mlsanity.com/paper/2609.27564/) (2026)

## Abstract (arXiv)

Inspired by neuronal diversity in the biological neural system, a plethora of studies proposed to design novel types of artificial neurons and introduce neuronal diversity into artificial neural networks. Recently proposed quadratic neuron, which replaces the inner-product operation in conventional neurons with a quadratic one, have achieved great success in many essential tasks. Despite the promising results of quadratic neurons, there is still an unresolved issue: textitIs the superior performance of quadratic networks simply due to the increased parameters or due to the intrinsic expressive capability? Without clarifying this issue, the performance of quadratic networks is always suspicious. Additionally, resolving this issue is reduced to finding killer applications of quadratic networks. In this paper, with theoretical and empirical studies, we show that quadratic networks enjoy parametric efficiency, thereby confirming that the superior performance of quadratic networks is due to the intrinsic expressive capability. This intrinsic expressive ability comes from that quadratic neurons can easily represent nonlinear interaction, while it is hard for conventional neurons. Theoretically, we derive the approximation efficiency of the quadratic network over conventional ones in terms of real space and manifolds. Moreover, from the perspective of the Barron space, we demonstrate that there exists a functional space whose functions can be approximated by quadratic networks in a dimension-free error, but the approximation error of conventional networks is dependent on dimensions. Empirically, experimental results on synthetic data, classic benchmarks, and real-world applications show that quadratic models broadly enjoy parametric efficiency, and the gain of efficiency depends on the task.