# Intrinsic dimension estimation for locally undersampled data

arXiv: https://arxiv.org/abs/1906.07670 · HTML: https://mlsanity.com/paper/1906.07670/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 10 of 10)

- [Detecting Adversarial Examples from Sensitivity Inconsistency of Spatial-Transform Domain](https://mlsanity.com/paper/2103.04302/) (2021)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation](https://mlsanity.com/paper/2303.05161/) (2023)
- [Scale adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification](https://mlsanity.com/paper/2405.15132/) (2024)
- [Unsupervised detection of semantic correlations in big data](https://mlsanity.com/paper/2411.02126/) (2024)
- [Manifold Hypothesis in Data Analysis: Double Geometrically-Probabilistic Approach to Manifold Dimension Estimation](https://mlsanity.com/paper/2107.03903/) (2021)
- [Estimating Dataset Dimension via Singular Metrics under the Manifold Hypothesis: Application to Inverse Problems](https://mlsanity.com/paper/2507.07291/) (2025)
- [MESS: Manifold Embedding Motivated Super Sampling](https://mlsanity.com/paper/2107.06566/) (2021)
- [Geometric Regularization from Overparameterization](https://mlsanity.com/paper/2202.09276/) (2022)
- [Density Estimation via Binless Multidimensional Integration](https://mlsanity.com/paper/2407.08094/) (2024)

## Abstract (arXiv)

High-dimensional data are ubiquitous in contemporary science and finding methods to compress them is one of the primary goals of machine learning. Given a dataset lying in a high-dimensional space (in principle hundreds to several thousands of dimensions), it is often useful to project it onto a lower-dimensional manifold, without loss of information. Identifying the minimal dimension of such manifold is a challenging problem known in the literature as intrinsic dimension estimation (IDE). Traditionally, most IDE algorithms are either based on multiscale principal component analysis (PCA) or on the notion of correlation dimension (and more in general on k-nearest-neighbors distances). These methods are affected, in different ways, by a severe curse of dimensionality. In particular, none of the existing algorithms can provide accurate ID estimates in the extreme locally undersampled regime, i.e. in the limit where the number of samples in any local patch of the manifold is less than (or of the same order of) the ID of the dataset. Here we introduce a new ID estimator that leverages on simple properties of the tangent space of a manifold to overcome these shortcomings. The method is based on the full correlation integral, going beyond the limit of small radius used for the estimation of the correlation dimension. Our estimator alleviates the extreme undersampling problem, intractable with other methods. Based on this insight, we explore a multiscale generalization of the algorithm. We show that it is capable of (i) identifying multiple dimensionalities in a dataset, and (ii) providing accurate estimates of the ID of extremely curved manifolds. In particular, we test the method on manifolds generated from global transformations of high-contrast images, relevant for invariant object recognition and considered a challenge for state-of-the-art ID estimators.