# Augmenting Deep Classifiers with Polynomial Neural Networks

arXiv: https://arxiv.org/abs/2104.07916 · HTML: https://mlsanity.com/paper/2104.07916/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 72 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)
- [The Shattered Gradients Problem: If resnets are the answer, then what is the question?](https://mlsanity.com/paper/1702.08591/) (2017)
- [Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks](https://mlsanity.com/paper/1810.12348/) (2018)
- [A2-Nets: Double Attention Networks](https://mlsanity.com/paper/1810.11579/) (2018)

## Built on by (most cited first, 21 of 21)

- [Polynormer: Polynomial-Expressive Graph Transformer in Linear Time](https://mlsanity.com/paper/2403.01232/) (2024)
- [Regularization of polynomial networks for image recognition](https://mlsanity.com/paper/2303.13896/) (2023)
- [Multilinear Operator Networks](https://mlsanity.com/paper/2401.17992/) (2024)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study](https://mlsanity.com/paper/2209.07736/) (2022)
- [One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks](https://mlsanity.com/paper/2303.06316/) (2023)
- [The Alexander-Hirschowitz theorem for neurovarieties](https://mlsanity.com/paper/2511.19703/) (2025)
- [Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2204.01707/) (2022)
- [Deep Tree Tensor Networks](https://mlsanity.com/paper/2502.09928/) (2025)
- [Sound and Complete Verification of Polynomial Networks](https://mlsanity.com/paper/2209.07235/) (2022)
- [PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding](https://mlsanity.com/paper/2602.01322/) (2026)
- [Muon Learns More Robust and Transferable Features than Adam](https://mlsanity.com/paper/2606.09658/) (2026)
- [PNeRV: A Polynomial Neural Representation for Videos](https://mlsanity.com/paper/2406.19299/) (2024)
- [Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation](https://mlsanity.com/paper/2406.02336/) (2024)
- [AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch](https://mlsanity.com/paper/2510.19368/) (2025)
- [Spectral Higher-Order Neural Networks](https://mlsanity.com/paper/2603.28420/) (2026)
- [NeuronSeek: On Stability and Expressivity of Task-driven Neurons](https://mlsanity.com/paper/2506.15715/) (2025)
- [(MPO)2: Multivariate Polynomial Optimization based on Matrix Product Operators](https://mlsanity.com/paper/2607.15916/) (2026)
- [Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds](https://mlsanity.com/paper/2607.19042/) (2026)
- [TNLearn: An Open Source Python Package for Task-based Neurons](https://mlsanity.com/paper/2609.27564/) (2026)

## Abstract (arXiv)

Deep neural networks have been the driving force behind the success in classification tasks, e.g., object and audio recognition. Impressive results and generalization have been achieved by a variety of recently proposed architectures, the majority of which are seemingly disconnected. In this work, we cast the study of deep classifiers under a unifying framework. In particular, we express state-of-the-art architectures (e.g., residual and non-local networks) in the form of different degree polynomials of the input. Our framework provides insights on the inductive biases of each model and enables natural extensions building upon their polynomial nature. The efficacy of the proposed models is evaluated on standard image and audio classification benchmarks. The expressivity of the proposed models is highlighted both in terms of increased model performance as well as model compression. Lastly, the extensions allowed by this taxonomy showcase benefits in the presence of limited data and long-tailed data distributions. We expect this taxonomy to provide links between existing domain-specific architectures. The source code is available at urlhttps://github.com/grigorisg9gr/polynomials-for-augmenting-NNs.