# Polynomial Regression As an Alternative to Neural Nets

arXiv: https://arxiv.org/abs/1806.06850 · HTML: https://mlsanity.com/paper/1806.06850/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 15 of 15)

- [Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing](https://mlsanity.com/paper/2210.04153/) (2022)
- [Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression](https://mlsanity.com/paper/2102.03865/) (2021)
- [Stimulative Training++: Go Beyond The Performance Limits of Residual Networks](https://mlsanity.com/paper/2305.02507/) (2023)
- [NN2Poly: A polynomial representation for deep feed-forward artificial neural networks](https://mlsanity.com/paper/2112.11397/) (2021)
- [Activation Functions Not To Active: A Plausible Theory on Interpreting Neural Networks](https://mlsanity.com/paper/2305.00663/) (2023)
- [A hybrid machine learning framework for clad characteristics prediction in metal additive manufacturing](https://mlsanity.com/paper/2307.01872/) (2023)
- [Normalized Augmented Inverse Probability Weighting with Neural Network Predictions](https://mlsanity.com/paper/2108.01768/) (2021)
- [Augmenting data-driven models for energy systems through feature engineering: A Python framework for feature engineering](https://mlsanity.com/paper/2301.01720/) (2023)
- [Privacy-Preserving 3-Layer Neural Network Training](https://mlsanity.com/paper/2308.09531/) (2023)
- [Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs](https://mlsanity.com/paper/2101.00674/) (2021)
- [Bagged Polynomial Regression and Neural Networks](https://mlsanity.com/paper/2205.08609/) (2022)
- [Data-driven discovery of Green's functions](https://mlsanity.com/paper/2210.16016/) (2022)
- [Branch and Bound to Assess Stability of Regression Coefficients in Uncertain Models](https://mlsanity.com/paper/2408.09634/) (2024)
- [Orbit Determination through Cosmic Microwave Background Radiation](https://mlsanity.com/paper/2504.02196/) (2025)
- [Simultaneous Optimization of Efficiency and Degradation in Tunable HTL-Free Perovskite Solar Cells with MWCNT-Integrated Back Contact Using a Machine Learning-Derived Polynomial Regressor](https://mlsanity.com/paper/2505.18693/) (2025)

## Abstract (arXiv)

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.