# Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection

arXiv: https://arxiv.org/abs/2204.01707 · HTML: https://mlsanity.com/paper/2204.01707/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 83 references that are in the map.
- First posted on arXiv 2 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition](https://mlsanity.com/paper/1903.06150/) (2019)
- [On the Computational Efficiency of Training Neural Networks](https://mlsanity.com/paper/1410.1141/) (2014)
- [Generative Adversarial Active Learning for Unsupervised Outlier Detection](https://mlsanity.com/paper/1809.10816/) (2018)
- [Deep Polynomial Neural Networks](https://mlsanity.com/paper/2006.13026/) (2020)
- [Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition](https://mlsanity.com/paper/1807.09915/) (2018)
- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [Low-rank Bilinear Pooling for Fine-Grained Classification](https://mlsanity.com/paper/1611.05109/) (2016)
- [Pi-nets: Deep Polynomial Neural Networks](https://mlsanity.com/paper/2003.03828/) (2020)
- [Non-linear Convolution Filters for CNN-based Learning](https://mlsanity.com/paper/1708.07038/) (2017)
- [Quadratic Autoencoder (Q-AE) for Low-dose CT Denoising](https://mlsanity.com/paper/1901.05593/) (2019)
- [Universal Approximation with Quadratic Deep Networks](https://mlsanity.com/paper/1808.00098/) (2018)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [DASVDD: Deep Autoencoding Support Vector Data Descriptor for Anomaly Detection](https://mlsanity.com/paper/2106.05410/) (2021)
- [QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration](https://mlsanity.com/paper/2204.01701/) (2022)
- [On a Sparse Shortcut Topology of Artificial Neural Networks](https://mlsanity.com/paper/1811.09003/) (2018)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [A New Type of Neurons for Machine Learning](https://mlsanity.com/paper/1704.08362/) (2017)
- [Blind Denoising Autoencoder](https://mlsanity.com/paper/1912.07358/) (2019)
- [adVAE: A self-adversarial variational autoencoder with Gaussian anomaly prior knowledge for anomaly detection](https://mlsanity.com/paper/1903.00904/) (2019)
- [Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement](https://mlsanity.com/paper/2112.05036/) (2021)
- [Deep Heterogeneous Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1812.06610/) (2018)

## Built on by (most cited first, 5 of 5)

- [Attention-embedded Quadratic Network (Qttention) for Effective and Interpretable Bearing Fault Diagnosis](https://mlsanity.com/paper/2206.00390/) (2022)
- [Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks](https://mlsanity.com/paper/2301.09245/) (2023)
- [ASBench: Image Anomalies Synthesis Benchmark for Anomaly Detection](https://mlsanity.com/paper/2510.07927/) (2025)
- [Retinal Vessel Segmentation via Neuron Programming](https://mlsanity.com/paper/2411.11110/) (2024)
- [No More DeLuLu: Physics-Inspired Kernel Networks for Geometrically-Grounded Neural Computation](https://mlsanity.com/paper/2603.12276/) (2026)

## Abstract (arXiv)

Inspired by the complexity and diversity of biological neurons, a quadratic neuron is proposed to replace the inner product in the current neuron with a simplified quadratic function. Employing such a novel type of neurons offers a new perspective on developing deep learning. When analyzing quadratic neurons, we find that there exists a function such that a heterogeneous network can approximate it well with a polynomial number of neurons but a purely conventional or quadratic network needs an exponential number of neurons to achieve the same level of error. Encouraged by this inspiring theoretical result on heterogeneous networks, we directly integrate conventional and quadratic neurons in an autoencoder to make a new type of heterogeneous autoencoders. To our best knowledge, it is the first heterogeneous autoencoder that is made of different types of neurons. Next, we apply the proposed heterogeneous autoencoder to unsupervised anomaly detection for tabular data and bearing fault signals. The anomaly detection faces difficulties such as data unknownness, anomaly feature heterogeneity, and feature unnoticeability, which is suitable for the proposed heterogeneous autoencoder. Its high feature representation ability can characterize a variety of anomaly data (heterogeneity), discriminate the anomaly from the normal (unnoticeability), and accurately learn the distribution of normal samples (unknownness). Experiments show that heterogeneous autoencoders perform competitively compared to other state-of-the-art models.