# Scale adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification

arXiv: https://arxiv.org/abs/2405.15132 · HTML: https://mlsanity.com/paper/2405.15132/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 65 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 17 of 17)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Scikit-dimension: a Python package for intrinsic dimension estimation](https://mlsanity.com/paper/2109.02596/) (2021)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [Intrinsic Dimensionality Estimation within Tight Localities: A Theoretical and Experimental Analysis](https://mlsanity.com/paper/2209.14475/) (2022)
- [Intrinsic Dimension Estimation Using Wasserstein Distances](https://mlsanity.com/paper/2106.04018/) (2021)
- [Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff](https://mlsanity.com/paper/2305.19008/) (2023)
- [Automatic topography of high-dimensional data sets by non-parametric Density Peak clustering](https://mlsanity.com/paper/1802.10549/) (2018)
- [Low-rank Characteristic Tensor Density Estimation Part I: Foundations](https://mlsanity.com/paper/2008.12315/) (2020)
- [Intrinsic dimension estimation for locally undersampled data](https://mlsanity.com/paper/1906.07670/) (2019)
- [Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation](https://mlsanity.com/paper/2204.00930/) (2022)
- [Posterior Contraction Rates for Mat'ern Gaussian Processes on Riemannian Manifolds](https://mlsanity.com/paper/2309.10918/) (2023)
- [Intrinsic dimension estimation for discrete metrics](https://mlsanity.com/paper/2207.09688/) (2022)
- [Dimension Estimation Using Random Connection Models](https://mlsanity.com/paper/1711.02876/) (2017)

## Built on by (most cited first, 3 of 3)

- [Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice](https://mlsanity.com/paper/2501.00982/) (2025)
- [A general framework for adaptive nonparametric dimensionality reduction](https://mlsanity.com/paper/2511.09486/) (2025)
- [Psychosis involves a deficit of information compression in connected speech](https://mlsanity.com/paper/2609.15522/) (2026)

## Abstract (arXiv)

The Intrinsic Dimension (ID) is a key concept in unsupervised learning and feature selection, as it is a lower bound to the number of variables which are necessary to describe a system. However, in almost any real-world dataset the ID depends on the scale at which the data are analysed. Quite typically at a small scale, the ID is very large, as the data are affected by measurement errors. At large scale, the ID can also appear erroneously large, due to the curvature and the topology of the manifold containing the data. In this work, we introduce an automatic protocol to select the sweet spot, namely the correct range of scales in which the ID is meaningful and useful. This protocol is based on imposing that for distances smaller than the correct scale the density of the data is constant. In the presented framework, to estimate the density it is necessary to know the ID, therefore, this condition is imposed self-consistently. We illustrate the usefulness and robustness of this procedure to noise by benchmarks on artificial and real-world datasets.