# On a Sparse Shortcut Topology of Artificial Neural Networks

arXiv: https://arxiv.org/abs/1811.09003 · HTML: https://mlsanity.com/paper/1811.09003/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2018.

## Built on by (most cited first, 20 of 20)

- [CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising](https://mlsanity.com/paper/2202.13517/) (2022)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [CTLformer: A Hybrid Denoising Model Combining Convolutional Layers and Self-Attention for Enhanced CT Image Reconstruction](https://mlsanity.com/paper/2505.12203/) (2025)
- [Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks](https://mlsanity.com/paper/2301.09245/) (2023)
- [One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks](https://mlsanity.com/paper/2303.06316/) (2023)
- [Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2204.01707/) (2022)
- [Retinex Image Enhancement Based on Sequential Decomposition With a Plug-and-Play Framework](https://mlsanity.com/paper/2210.05436/) (2022)
- [Neural Network Gaussian Processes by Increasing Depth](https://mlsanity.com/paper/2108.12862/) (2021)
- [No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks](https://mlsanity.com/paper/2405.02369/) (2024)
- [Enhancing the Resilience of Graph Neural Networks to Topological Perturbations in Sparse Graphs](https://mlsanity.com/paper/2406.03097/) (2024)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [Layer-wise Geometric Approximation Rates for Deep Networks](https://mlsanity.com/paper/2604.20219/) (2026)
- [SQ-Swin: a Pretrained Siamese Quadratic Swin Transformer for Lettuce Browning Prediction](https://mlsanity.com/paper/2209.07683/) (2022)
- [Uncertainty Aware Neural Network from Similarity and Sensitivity](https://mlsanity.com/paper/2304.14925/) (2023)
- [Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels](https://mlsanity.com/paper/2511.05585/) (2025)
- [LoMAE: Low-level Vision Masked Autoencoders for Low-dose CT Denoising](https://mlsanity.com/paper/2310.12405/) (2023)
- [Neural Network Approximation: A View from Polytope Decomposition](https://mlsanity.com/paper/2601.18264/) (2026)
- [Enhancing Layer Interaction Using Key-Correlated Layer Attention](https://mlsanity.com/paper/2606.28405/) (2026)
- [TNLearn: An Open Source Python Package for Task-based Neurons](https://mlsanity.com/paper/2609.27564/) (2026)

## Abstract (arXiv)

In established network architectures, shortcut connections are often used to take the outputs of earlier layers as additional inputs to later layers. Despite the extraordinary effectiveness of shortcuts, there remain open questions on the mechanism and characteristics. For example, why are shortcuts powerful? Why do shortcuts generalize well? In this paper, we investigate the expressivity and generalizability of a novel sparse shortcut topology. First, we demonstrate that this topology can empower a one-neuron-wide deep network to approximate any univariate continuous function. Then, we present a novel width-bounded universal approximator in contrast to depth-bounded universal approximators and extend the approximation result to a family of equally competent networks. Furthermore, with generalization bound theory, we show that the proposed shortcut topology enjoys excellent generalizability. Finally, we corroborate our theoretical analyses by comparing the proposed topology with popular architectures, including ResNet and DenseNet, on well-known benchmarks and perform a saliency map analysis to interpret the proposed topology. Our work helps enhance the understanding of the role of shortcuts and suggests further opportunities to innovate neural architectures.