# White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?

arXiv: https://arxiv.org/abs/2311.13110 · HTML: https://mlsanity.com/paper/2311.13110/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2023.

## Built on by (most cited first, 23 of 23)

- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [A Law of Next-Token Prediction in Large Language Models](https://mlsanity.com/paper/2408.13442/) (2024)
- [A Global Geometric Analysis of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2406.01909/) (2024)
- [Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction](https://mlsanity.com/paper/2412.17810/) (2024)
- [Scaling White-Box Transformers for Vision](https://mlsanity.com/paper/2405.20299/) (2024)
- [Attention-Only Transformers via Unrolled Subspace Denoising](https://mlsanity.com/paper/2506.03790/) (2025)
- [Large Vision-Language Models Get Lost in Attention](https://mlsanity.com/paper/2605.05668/) (2026)
- [DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction](https://mlsanity.com/paper/2409.10504/) (2024)
- [Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training](https://mlsanity.com/paper/2511.07372/) (2025)
- [Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle](https://mlsanity.com/paper/2511.00907/) (2025)
- [FAST: Factorizable Attention for Speeding up Transformers](https://mlsanity.com/paper/2402.07901/) (2024)
- [Improving Neuron-level Interpretability with White-box Language Models](https://mlsanity.com/paper/2410.16443/) (2024)
- [White-Box Diffusion Transformer for single-cell RNA-seq generation](https://mlsanity.com/paper/2411.06785/) (2024)
- [ESS-ReduNet: Enhancing Subspace Separability of ReduNet via Dynamic Expansion with Bayesian Inference](https://mlsanity.com/paper/2411.17961/) (2024)
- [Towards White-Box Deep Wireless Sensing](https://mlsanity.com/paper/2507.21799/) (2025)
- [Transformer Based Linear Attention with Optimized GPU Kernel Implementation](https://mlsanity.com/paper/2510.21956/) (2025)
- [Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So](https://mlsanity.com/paper/2608.10251/) (2026)
- [Way to Build Native AI-driven 6G Air Interface: Principles, Roadmap, and Outlook](https://mlsanity.com/paper/2508.15277/) (2025)
- [PRISM: Deriving a White-Box Transformer as a Signal-Noise Decomposition Operator via Maximum Coding Rate Reduction](https://mlsanity.com/paper/2601.15540/) (2026)
- [Interpretable and Sparse Linear Attention with Decoupled Membership-Subspace Modeling via MCR2 Objective](https://mlsanity.com/paper/2601.17042/) (2026)
- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)
- [On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation](https://mlsanity.com/paper/2609.21001/) (2026)
- [Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures](https://mlsanity.com/paper/2610.04413/) (2026)

## Abstract (arXiv)

In this paper, we contend that a natural objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a low-dimensional Gaussian mixture supported on incoherent subspaces. The goodness of such a representation can be evaluated by a principled measure, called sparse rate reduction, that simultaneously maximizes the intrinsic information gain and extrinsic sparsity of the learned representation. From this perspective, popular deep network architectures, including transformers, can be viewed as realizing iterative schemes to optimize this measure. Particularly, we derive a transformer block from alternating optimization on parts of this objective: the multi-head self-attention operator compresses the representation by implementing an approximate gradient descent step on the coding rate of the features, and the subsequent multi-layer perceptron sparsifies the features. This leads to a family of white-box transformer-like deep network architectures, named CRATE, which are mathematically fully interpretable. We show, by way of a novel connection between denoising and compression, that the inverse to the aforementioned compressive encoding can be realized by the same class of CRATE architectures. Thus, the so-derived white-box architectures are universal to both encoders and decoders. Experiments show that these networks, despite their simplicity, indeed learn to compress and sparsify representations of large-scale real-world image and text datasets, and achieve performance very close to highly engineered transformer-based models: ViT, MAE, DINO, BERT, and GPT2. We believe the proposed computational framework demonstrates great potential in bridging the gap between theory and practice of deep learning, from a unified perspective of data compression. Code is available at: https://ma-lab-berkeley.github.io/CRATE .