# FractalNet: Ultra-Deep Neural Networks without Residuals

arXiv: https://arxiv.org/abs/1605.07648 · HTML: https://mlsanity.com/paper/1605.07648/

## Key facts

- Cited by 216 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2016.

## Built on by (most cited first, 30 of 216)

- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Benchmarking Detection Transfer Learning with Vision Transformers](https://mlsanity.com/paper/2111.11429/) (2021)
- [SwinTrack: A Simple and Strong Baseline for Transformer Tracking](https://mlsanity.com/paper/2112.00995/) (2021)
- [Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks](https://mlsanity.com/paper/2112.01522/) (2021)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [MOOD: Multi-level Out-of-distribution Detection](https://mlsanity.com/paper/2104.14726/) (2021)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [S2-MLP: Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2106.07477/) (2021)
- [SODA: Bottleneck Diffusion Models for Representation Learning](https://mlsanity.com/paper/2311.17901/) (2023)
- [Diffusion Models as Masked Autoencoders](https://mlsanity.com/paper/2304.03283/) (2023)
- [Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance](https://mlsanity.com/paper/2403.05231/) (2024)
- [Encoding physics to learn reaction-diffusion processes](https://mlsanity.com/paper/2106.04781/) (2021)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [Bilateral Propagation Network for Depth Completion](https://mlsanity.com/paper/2403.11270/) (2024)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution](https://mlsanity.com/paper/2211.11436/) (2022)
- [Fractal Generative Models](https://mlsanity.com/paper/2502.17437/) (2025)
- [Reflash Dropout in Image Super-Resolution](https://mlsanity.com/paper/2112.12089/) (2021)
- [Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?](https://mlsanity.com/paper/2109.05422/) (2021)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Illuminating Diverse Neural Cellular Automata for Level Generation](https://mlsanity.com/paper/2109.05489/) (2021)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)

## Abstract (arXiv)

We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.