# Dimensionality compression and expansion in Deep Neural Networks

arXiv: https://arxiv.org/abs/1906.00443 · HTML: https://mlsanity.com/paper/1906.00443/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2019.

## Built on by (most cited first, 30 of 38)

- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [Learning sparse features can lead to overfitting in neural networks](https://mlsanity.com/paper/2206.12314/) (2022)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Outlier Dimensions Encode Task-Specific Knowledge](https://mlsanity.com/paper/2310.17715/) (2023)
- [Intermediate Layer Classifiers for OOD generalization](https://mlsanity.com/paper/2504.05461/) (2025)
- [A Global Geometric Analysis of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2406.01909/) (2024)
- [Relative stability toward diffeomorphisms indicates performance in deep nets](https://mlsanity.com/paper/2105.02468/) (2021)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)
- [Origami in N dimensions: How feed-forward networks manufacture linear separability](https://mlsanity.com/paper/2203.11355/) (2022)
- [Relating Regularization and Generalization through the Intrinsic Dimension of Activations](https://mlsanity.com/paper/2211.13239/) (2022)
- [A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs](https://mlsanity.com/paper/2105.01867/) (2021)
- [Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation](https://mlsanity.com/paper/2303.05161/) (2023)
- [How deep convolutional neural networks lose spatial information with training](https://mlsanity.com/paper/2210.01506/) (2022)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Stochastic Forward-Forward Learning through Representational Dimensionality Compression](https://mlsanity.com/paper/2505.16649/) (2025)
- [Rare Text Semantics Were Always There in Your Diffusion Transformer](https://mlsanity.com/paper/2510.03886/) (2025)
- [Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs](https://mlsanity.com/paper/2110.11400/) (2021)
- [CNNs Avoid Curse of Dimensionality by Learning on Patches](https://mlsanity.com/paper/2205.10760/) (2022)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations](https://mlsanity.com/paper/2407.17754/) (2024)
- [Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimension](https://mlsanity.com/paper/2501.02364/) (2025)
- [Intermediate Representations are Strong AI-Generated Image Detectors](https://mlsanity.com/paper/2605.04358/) (2026)
- [The Foes of Neural Network's Data Efficiency Among Unnecessary Input Dimensions](https://mlsanity.com/paper/2107.06409/) (2021)
- [Exploring Learned Representations of Neural Networks with Principal Component Analysis](https://mlsanity.com/paper/2309.15328/) (2023)
- [On original and latent space connectivity in deep neural networks](https://mlsanity.com/paper/2311.06816/) (2023)
- [Neural Network Approximation for Pessimistic Offline Reinforcement Learning](https://mlsanity.com/paper/2312.11863/) (2023)

## Abstract (arXiv)

Datasets such as images, text, or movies are embedded in high-dimensional spaces. However, in important cases such as images of objects, the statistical structure in the data constrains samples to a manifold of dramatically lower dimensionality. Learning to identify and extract task-relevant variables from this embedded manifold is crucial when dealing with high-dimensional problems. We find that neural networks are often very effective at solving this task and investigate why. To this end, we apply state-of-the-art techniques for intrinsic dimensionality estimation to show that neural networks learn low-dimensional manifolds in two phases: first, dimensionality expansion driven by feature generation in initial layers, and second, dimensionality compression driven by the selection of task-relevant features in later layers. We model noise generated by Stochastic Gradient Descent and show how this noise balances the dimensionality of neural representations by inducing an effective regularization term in the loss. We highlight the important relationship between low-dimensional compressed representations and generalization properties of the network. Our work contributes by shedding light on the success of deep neural networks in disentangling data in high-dimensional space while achieving good generalization. Furthermore, it invites new learning strategies focused on optimizing measurable geometric properties of learned representations, beginning with their intrinsic dimensionality.