# Neural Normalized Compression Distance and the Disconnect Between Compression and Classification

arXiv: https://arxiv.org/abs/2410.15280 · HTML: https://mlsanity.com/paper/2410.15280/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 23 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Lempel-Ziv Jaccard Distance, an Effective Alternative to Ssdeep and Sdhash](https://mlsanity.com/paper/1708.03346/) (2017)
- [A New Burrows Wheeler Transform Markov Distance](https://mlsanity.com/paper/1912.13046/) (2019)

## Built on by (most cited first, 3 of 3)

- [Context Steering: A New Paradigm for Compression-based Embeddings by Synthesizing Relevant Information Features](https://mlsanity.com/paper/2508.14780/) (2025)
- [Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection](https://mlsanity.com/paper/2607.21752/) (2026)
- [An Introduction to Compression-Based Machine Learning](https://mlsanity.com/paper/2609.21309/) (2026)

## Abstract (arXiv)

It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. In doing so, we find that classification accuracy is not predictable by compression rate alone, among other empirical aberrations not predicted by current understanding. Our results imply that our intuition on what it means for a neural network to ``compress'' and what is needed for effective classification are not yet well understood.