# Compressible Softmax-Attended Language under Incompressible Attention

arXiv: https://arxiv.org/abs/2604.04384 · HTML: https://mlsanity.com/paper/2604.04384/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 4 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Built on by (most cited first, 1 of 1)

- [Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention](https://mlsanity.com/paper/2608.28150/) (2026)

## Abstract (arXiv)

Softmax attention defines an interaction through dh head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field tildeE reaches 90% of its variance in 2--11 singular components. The learned interaction matrix WQT WK needs 38--75 components for the same threshold out of dh ∈ 64, 128. The spectral gap is 5--25× in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.