mlsanityData to 7 Oct 2026

Rethinking Intrinsic Dimension Estimation in Neural Representations

arXiv 2604.20276 ↗First posted 22 Apr 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The analysis of neural representation has become an integral part of research aiming to better understand the inner workings of neural networks. While there are many different approaches to investigate neural representations, an important line of research has focused on doing so through the lens of intrinsic dimensions (IDs). Although this perspective has provided valuable insights and stimulated substantial follow-up research, important limitations of this approach have remained largely unaddressed. In this paper, we highlight a crucial discrepancy between theory and practice of IDs in neural representations, theoretically and empirically showing that common ID estimators are, in fact, not tracking the true underlying ID of the representation. We contrast this negative result with an investigation of the underlying factors that may drive commonly reported ID-related results on neural representation in the literature. Building on these insights, we offer a new perspective on ID estimation in neural representations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 86 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

38 of 86 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  7. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  8. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  9. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  10. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  11. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  12. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
26 more
  1. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  2. 2020The Lipschitz Constant of Self-Attentioncited by 171
  3. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  4. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 136
  5. 2023The geometry of hidden representations of large transformer modelscited by 108
  6. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  7. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  8. 2021Scikit-dimension: a Python package for intrinsic dimension estimationcited by 68
  9. 2022Verifying the Union of Manifolds Hypothesis for Image Datacited by 62
  10. 2018On the Intrinsic Dimensionality of Image Representationscited by 57
  11. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  12. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  13. 2022DADApy: Distance-based Analysis of DAta-manifolds in Pythoncited by 22
  14. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  15. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  16. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  17. 2025Token embeddings violate the manifold hypothesiscited by 14
  18. 2024The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Imagescited by 13
  19. 2025A Survey of Dimension Estimation Methodscited by 11
  20. 2025Unlocking Pretrained Vision Transformers for Time Series Classificationcited by 6
  21. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  22. 2022Internal Representations of Vision Models Through the Lens of Frames on Data Manifoldscited by 3
  23. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  24. 2025Adjustment for Confounding using Pre-Trained Representationscited by 3
  25. 2013On the Estimation of Pointwise Dimensioncited by 1
  26. 2024Pre-processing and Compression: Understanding Hidden Representation Refinement Across Imaging Domains via Intrinsic Dimensioncited by 1

Built on by

Not cited yet.