# Pi-nets: Deep Polynomial Neural Networks

arXiv: https://arxiv.org/abs/2003.03828 · HTML: https://mlsanity.com/paper/2003.03828/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2020.

## Built on by (most cited first, 30 of 39)

- [Polynormer: Polynomial-Expressive Graph Transformer in Linear Time](https://mlsanity.com/paper/2403.01232/) (2024)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Scalable Interpretability via Polynomials](https://mlsanity.com/paper/2205.14108/) (2022)
- [Polynomial Neural Fields for Subband Decomposition and Manipulation](https://mlsanity.com/paper/2302.04862/) (2023)
- [QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration](https://mlsanity.com/paper/2204.01701/) (2022)
- [Geometry of Polynomial Neural Networks](https://mlsanity.com/paper/2402.00949/) (2024)
- [The Spectral Bias of Polynomial Neural Networks](https://mlsanity.com/paper/2202.13473/) (2022)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [Beyond Linear Probes: Dynamic Safety Monitoring for Language Models](https://mlsanity.com/paper/2509.26238/) (2025)
- [Tensor Component Analysis for Interpreting the Latent Space of GANs](https://mlsanity.com/paper/2111.11736/) (2021)
- [On the Ability of Graph Neural Networks to Model Interactions Between Vertices](https://mlsanity.com/paper/2211.16494/) (2022)
- [Multilinear Operator Networks](https://mlsanity.com/paper/2401.17992/) (2024)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study](https://mlsanity.com/paper/2209.07736/) (2022)
- [Power-Softmax: Towards Secure LLM Inference over Encrypted Data](https://mlsanity.com/paper/2410.09457/) (2024)
- [Personalizing Pre-trained Models](https://mlsanity.com/paper/2106.01499/) (2021)
- [3D human tongue reconstruction from single "in-the-wild" images](https://mlsanity.com/paper/2106.12302/) (2021)
- [Poly-NL: Linear Complexity Non-local Layers with Polynomials](https://mlsanity.com/paper/2107.02859/) (2021)
- [Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2204.01707/) (2022)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks](https://mlsanity.com/paper/2311.17956/) (2023)
- [Adversarial Audio Synthesis with Complex-valued Polynomial Networks](https://mlsanity.com/paper/2206.06811/) (2022)
- [The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theory](https://mlsanity.com/paper/2306.14753/) (2023)
- [A Training Framework for Optimal and Stable Training of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11589/) (2025)
- [PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding](https://mlsanity.com/paper/2602.01322/) (2026)
- [A Novel Convolutional Neural Network Architecture with a Continuous Symmetry](https://mlsanity.com/paper/2308.01621/) (2023)
- [On the Study of Sample Complexity for Polynomial Neural Networks](https://mlsanity.com/paper/2207.08896/) (2022)
- [Tackling Face Verification Edge Cases: In-Depth Analysis and Human-Machine Fusion Approach](https://mlsanity.com/paper/2304.08134/) (2023)
- [Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation](https://mlsanity.com/paper/2406.02336/) (2024)

## Abstract (arXiv)

Deep Convolutional Neural Networks (DCNNs) is currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to the careful selection of their building blocks (e.g., residual blocks, rectifiers, sophisticated normalization schemes, to mention but a few). In this paper, we propose Pi-Nets, a new class of DCNNs. Pi-Nets are polynomial neural networks, i.e., the output is a high-order polynomial of the input. Pi-Nets can be implemented using special kind of skip connections and their parameters can be represented via high-order tensors. We empirically demonstrate that Pi-Nets have better representation power than standard DCNNs and they even produce good results without the use of non-linear activation functions in a large battery of tasks and signals, i.e., images, graphs, and audio. When used in conjunction with activation functions, Pi-Nets produce state-of-the-art results in challenging tasks, such as image generation. Lastly, our framework elucidates why recent generative models, such as StyleGAN, improve upon their predecessors, e.g., ProGAN.