# Unsupervised detection of semantic correlations in big data

arXiv: https://arxiv.org/abs/2411.02126 · HTML: https://mlsanity.com/paper/2411.02126/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [A Survey on Methods and Theories of Quantized Neural Networks](https://mlsanity.com/paper/1808.04752/) (2018)
- [Emergence of a High-Dimensional Abstraction Phase in Language Transformers](https://mlsanity.com/paper/2405.15471/) (2024)
- [Bridging Information-Theoretic and Geometric Compression in Language Models](https://mlsanity.com/paper/2310.13620/) (2023)
- [The High-Dimensional Geometry of Binary Neural Networks](https://mlsanity.com/paper/1705.07199/) (2017)
- [Ranking the information content of distance measures](https://mlsanity.com/paper/2104.15079/) (2021)
- [DADApy: Distance-based Analysis of DAta-manifolds in Python](https://mlsanity.com/paper/2205.03373/) (2022)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [Intrinsic dimension estimation for locally undersampled data](https://mlsanity.com/paper/1906.07670/) (2019)
- [Phase transitions in the mini-batch size for sparse and dense two-layer neural networks](https://mlsanity.com/paper/2305.06435/) (2023)
- [Intrinsic dimension estimation for discrete metrics](https://mlsanity.com/paper/2207.09688/) (2022)
- [Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations](https://mlsanity.com/paper/2406.15812/) (2024)
- [What is the textitintrinsic dimension of your binary data? -- and how to compute it quickly](https://mlsanity.com/paper/2404.06326/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Intrinsic Dimension of Prompts in Internal Representations of Large Language Models](https://mlsanity.com/paper/2501.10573/) (2025)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)

## Abstract (arXiv)

In real-world data, information is stored in extremely large feature vectors. These variables are typically correlated due to complex interactions involving many features simultaneously. Such correlations qualitatively correspond to semantic roles and are naturally recognized by both the human brain and artificial neural networks. This recognition enables, for instance, the prediction of missing parts of an image or text based on their context. We present a method to detect these correlations in high-dimensional data represented as binary numbers. We estimate the binary intrinsic dimension of a dataset, which quantifies the minimum number of independent coordinates needed to describe the data, and is therefore a proxy of semantic complexity. The proposed algorithm is largely insensitive to the so-called curse of dimensionality, and can therefore be used in big data analysis. We test this approach identifying phase transitions in model magnetic systems and we then apply it to the detection of semantic correlations of images and text inside deep neural networks.