# Ranking the information content of distance measures

arXiv: https://arxiv.org/abs/2104.15079 · HTML: https://mlsanity.com/paper/2104.15079/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 50 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Low Data Drug Discovery with One-shot Learning](https://mlsanity.com/paper/1611.03199/) (2016)
- [Information estimation using nonparametric copulas](https://mlsanity.com/paper/1807.08018/) (2018)
- [The role of feature space in atomistic learning](https://mlsanity.com/paper/2009.02741/) (2020)

## Built on by (most cited first, 22 of 22)

- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [CryoBench: Diverse and challenging datasets for the heterogeneity problem in cryo-EM](https://mlsanity.com/paper/2408.05526/) (2024)
- [Universally Converging Representations of Matter Across Scientific Foundation Models](https://mlsanity.com/paper/2512.03750/) (2025)
- [Tensor-reduced atomic density representations](https://mlsanity.com/paper/2210.01705/) (2022)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance](https://mlsanity.com/paper/2411.00851/) (2024)
- [Multi-Way Representation Alignment](https://mlsanity.com/paper/2602.06205/) (2026)
- [Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations](https://mlsanity.com/paper/2601.03779/) (2026)
- [Unsupervised detection of semantic correlations in big data](https://mlsanity.com/paper/2411.02126/) (2024)
- [The Effect of Label Noise on the Information Content of Neural Representations](https://mlsanity.com/paper/2510.06401/) (2025)
- [Similarity of Processing Steps in Vision Model Representations](https://mlsanity.com/paper/2601.21621/) (2026)
- [Understanding Variational Autoencoders with Intrinsic Dimension and Information Imbalance](https://mlsanity.com/paper/2411.01978/) (2024)
- [Optimal transfer protocol by incremental layer defrosting](https://mlsanity.com/paper/2303.01429/) (2023)
- [Comparing the latent features of universal machine-learning interatomic potentials](https://mlsanity.com/paper/2512.05717/) (2025)
- [The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?](https://mlsanity.com/paper/2605.09352/) (2026)
- [Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networks](https://mlsanity.com/paper/2605.26973/) (2026)
- [Visual Instruction Tuning Aligns Modalities through Abstraction](https://mlsanity.com/paper/2606.03871/) (2026)
- [The limits of interpretability in multiple linear regression](https://mlsanity.com/paper/2606.16013/) (2026)
- [Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception](https://mlsanity.com/paper/2607.08321/) (2026)
- [The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers](https://mlsanity.com/paper/2608.25166/) (2026)

## Abstract (arXiv)

Real-world data typically contain a large number of features that are often heterogeneous in nature, relevance, and also units of measure. When assessing the similarity between data points, one can build various distance measures using subsets of these features. Using the fewest features but still retaining sufficient information about the system is crucial in many statistical learning approaches, particularly when data are sparse. We introduce a statistical test that can assess the relative information retained when using two different distance measures, and determine if they are equivalent, independent, or if one is more informative than the other. This in turn allows finding the most informative distance measure out of a pool of candidates. The approach is applied to find the most relevant policy variables for controlling the Covid-19 epidemic and to find compact yet informative representations of atomic structures, but its potential applications are wide ranging in many branches of science.