# A Deep Neural Network's Loss Surface Contains Every Low-dimensional Pattern

arXiv: https://arxiv.org/abs/1912.07559 · HTML: https://mlsanity.com/paper/1912.07559/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021)
- [TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation](https://mlsanity.com/paper/2503.04872/) (2025)
- [Multi-attacks: Many images + the same adversarial attack → many target labels](https://mlsanity.com/paper/2308.03792/) (2023)

## Abstract (arXiv)

The work "Loss Landscape Sightseeing with Multi-Point Optimization" (Skorokhodov and Burtsev, 2019) demonstrated that one can empirically find arbitrary 2D binary patterns inside loss surfaces of popular neural networks. In this paper we prove that: (i) this is a general property of deep universal approximators; and (ii) this property holds for arbitrary smooth patterns, for other dimensionalities, for every dataset, and any neural network that is sufficiently deep and wide. Our analysis predicts not only the existence of all such low-dimensional patterns, but also two other properties that were observed empirically: (i) that it is easy to find these patterns; and (ii) that they transfer to other data-sets (e.g. a test-set).