mlsanityData to 7 Oct 2026

Intrinsic dimension of data representations in deep neural networks

arXiv 1905.12784 ↗First posted 29 May 2019cs.LG stat.MLcited by 276

Not on the map. Why?

Abstract (arXiv)

Deep neural networks progressively transform their inputs across multiple processing layers. What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data-representations, i.e. the minimal number of parameters needed to describe a representation. We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer. Across layers, the ID first increases and then progressively decreases in the final layers. Remarkably, the ID of the last hidden layer predicts classification accuracy on the test set. These results can neither be found by linear dimensionality estimates (e.g., with principal component analysis), nor in representations that had been artificially linearized. They are neither found in untrained networks, nor in networks that are trained on randomized labels. This suggests that neural networks that can generalize are those that transform the data into low-dimensional, but not necessarily flat manifolds.

Key facts. Cited by 276 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 30; 2023: 35; 2024: 40; 2025: 61; 2026: 89.

276 papers.

Sort:

By area:

  1. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  3. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  4. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  5. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  6. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  7. 2023The geometry of hidden representations of large transformer modelscited by 108
  8. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  9. 2023BrainBERT: Self-supervised representation learning for intracranial recordingscited by 70
  10. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  11. 2022Verifying the Union of Manifolds Hypothesis for Image Datacited by 62
  12. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
38 more
  1. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  2. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  3. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  4. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  5. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  6. 2023Detecting Images Generated by Deep Diffusion Models using their Local Intrinsic Dimensionalitycited by 44
  7. 2024A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Modelscited by 41
  8. 2022Low Dimensional Invariant Embeddings for Universal Geometric Learningcited by 38
  9. 2021Rethinking Graph Auto-Encoder Models for Attributed Graph Clusteringcited by 37
  10. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  11. 2022LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihoodcited by 30
  12. 2025EEG Foundation Models: A Critical Review of Current Progress and Future Directionscited by 30
  13. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  14. 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
  15. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  16. 2024Your Transformer is Secretly Linearcited by 23
  17. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  18. 2022DADApy: Distance-based Analysis of DAta-manifolds in Pythoncited by 22
  19. 2023Complexity Matters: Rethinking the Latent Space for Generative Modelingcited by 19
  20. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  21. 2023Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learningcited by 18
  22. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  23. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  24. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  25. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  26. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
  27. 2024Memorization in deep learning: A surveycited by 14
  28. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  29. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  30. 2025The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Modelscited by 14
  31. 2025Intermediate Layer Classifiers for OOD generalizationcited by 14
  32. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representationscited by 14
  33. 2024The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Imagescited by 13
  34. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  35. 2021On the interplay between data structure and loss function in classification problemscited by 12
  36. 2021Relative stability toward diffeomorphisms indicates performance in deep netscited by 12
  37. 2021Neural Population Geometry Reveals the Role of Stochasticity in Robust Perceptioncited by 12
  38. 2022Topology and geometry of data manifold in deep learningcited by 12

50 most cited of 276 All 276 (JSON)