# Estimating the effective dimension of large biological datasets using Fisher separability analysis

arXiv: https://arxiv.org/abs/1901.06328 · HTML: https://mlsanity.com/paper/1901.06328/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2019.

## Built on by (most cited first, 19 of 19)

- [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://mlsanity.com/paper/2306.04723/) (2023)
- [Scikit-dimension: a Python package for intrinsic dimension estimation](https://mlsanity.com/paper/2109.02596/) (2021)
- [A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models](https://mlsanity.com/paper/2406.03537/) (2024)
- [LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood](https://mlsanity.com/paper/2206.14882/) (2022)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [Byte Pair Encoding for Symbolic Music](https://mlsanity.com/paper/2301.11975/) (2023)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Measuring the Intrinsic Dimension of Earth Representations](https://mlsanity.com/paper/2511.02101/) (2025)
- [Impact of time and note duration tokenizations on deep learning symbolic music modeling](https://mlsanity.com/paper/2310.08497/) (2023)
- [Relative intrinsic dimensionality is intrinsic to learning](https://mlsanity.com/paper/2311.07579/) (2023)
- [Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation](https://mlsanity.com/paper/2203.16687/) (2022)
- [Understanding the Structure of QM7b and QM9 Quantum Mechanical Datasets Using Unsupervised Learning](https://mlsanity.com/paper/2309.15130/) (2023)
- [A Universal Nearest-Neighbor Estimator for Intrinsic Dimensionality](https://mlsanity.com/paper/2603.10493/) (2026)
- [Data-Informed Model Complexity Metric for Optimizing Symbolic Regression Models](https://mlsanity.com/paper/2501.17372/) (2025)
- [A Novel Approach for Intrinsic Dimension Estimation](https://mlsanity.com/paper/2503.09485/) (2025)
- [Intrinsic Dimensionality as a Model-Free Measure of Class Imbalance](https://mlsanity.com/paper/2511.10475/) (2025)
- [Stratifying Reinforcement Learning with Signal Temporal Logic](https://mlsanity.com/paper/2604.04923/) (2026)
- [FiGuRO: Intrinsic Dimension Estimation for Multi-Modal Data](https://mlsanity.com/paper/2608.10857/) (2026)
- [Stochastic Separability of Embedding Manifolds](https://mlsanity.com/paper/2608.22874/) (2026)

## Abstract (arXiv)

Modern large-scale datasets are frequently said to be high-dimensional. However, their data point clouds frequently possess structures, significantly decreasing their intrinsic dimensionality (ID) due to the presence of clusters, points being located close to low-dimensional varieties or fine-grained lumping. We test a recently introduced dimensionality estimator, based on analysing the separability properties of data points, on several benchmarks and real biological datasets. We show that the introduced measure of ID has performance competitive with state-of-the-art measures, being efficient across a wide range of dimensions and performing better in the case of noisy samples. Moreover, it allows estimating the intrinsic dimension in situations where the intrinsic manifold assumption is not valid.