# Spectral Higher-Order Neural Networks

arXiv: https://arxiv.org/abs/2603.28420 · HTML: https://mlsanity.com/paper/2603.28420/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Simplicial Neural Networks](https://mlsanity.com/paper/2010.03633/) (2020)
- [Deep Polynomial Neural Networks](https://mlsanity.com/paper/2006.13026/) (2020)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [Machine learning in spectral domain](https://mlsanity.com/paper/2005.14436/) (2020)
- [Estimating Global Input Relevance and Enforcing Sparse Representations with a Scalable Spectral Neural Network Approach](https://mlsanity.com/paper/2406.01183/) (2024)
- [Spectral Architecture Search for Neural Network Models](https://mlsanity.com/paper/2504.00885/) (2025)

## Built on by (most cited first, 1 of 1)

- [Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds](https://mlsanity.com/paper/2607.19042/) (2026)

## Abstract (arXiv)

Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.