# QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks

arXiv: https://arxiv.org/abs/2311.17956 · HTML: https://mlsanity.com/paper/2311.17956/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 16 of 16)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions](https://mlsanity.com/paper/2207.14284/) (2022)
- [Pi-nets: Deep Polynomial Neural Networks](https://mlsanity.com/paper/2003.03828/) (2020)
- [QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration](https://mlsanity.com/paper/2204.01701/) (2022)
- [A New Type of Neurons for Machine Learning](https://mlsanity.com/paper/1704.08362/) (2017)

## Built on by (most cited first, 5 of 5)

- [QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation](https://mlsanity.com/paper/2405.03192/) (2024)
- [Automatic Screening for Children with Speech Disorder using Automatic Speech Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.11865/) (2024)
- [Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction](https://mlsanity.com/paper/2505.17999/) (2025)
- [Infinite-Dimensional Feature Interaction](https://mlsanity.com/paper/2405.13972/) (2024)
- [QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks](https://mlsanity.com/paper/2510.03276/) (2025)

## Abstract (arXiv)

Recent progress in computer vision-oriented neural network designs is mostly driven by capturing high-order neural interactions among inputs and features. And there emerged a variety of approaches to accomplish this, such as Transformers and its variants. However, these interactions generate a large amount of intermediate state and/or strong data dependency, leading to considerable memory consumption and computing cost, and therefore compromising the overall runtime performance. To address this challenge, we rethink the high-order interactive neural network design with a quadratic computing approach. Specifically, we propose QuadraNet -- a comprehensive model design methodology from neuron reconstruction to structural block and eventually to the overall neural network implementation. Leveraging quadratic neurons' intrinsic high-order advantages and dedicated computation optimization schemes, QuadraNet could effectively achieve optimal cognition and computation performance. Incorporating state-of-the-art hardware-aware neural architecture search and system integration techniques, QuadraNet could also be well generalized in different hardware constraint settings and deployment scenarios. The experiment shows thatQuadraNet achieves up to 1.5× throughput, 30% less memory footprint, and similar cognition performance, compared with the state-of-the-art high-order approaches.