# Compressing Large Language Models with PCA Without Performance Loss

arXiv: https://arxiv.org/abs/2508.04307 · HTML: https://mlsanity.com/paper/2508.04307/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 20 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 5 of 5)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [On the Downstream Performance of Compressed Word Embeddings](https://mlsanity.com/paper/1909.01264/) (2019)
- [Single Training Dimension Selection for Word Embedding with PCA](https://mlsanity.com/paper/1909.01761/) (2019)
- [Knowledge Base Index Compression via Dimensionality and Precision Reduction](https://mlsanity.com/paper/2204.02906/) (2022)

## Built on by (most cited first, 1 of 1)

- [Rethinking the Harmonic Loss via Non-Euclidean Distance Layers](https://mlsanity.com/paper/2603.10225/) (2026)

## Abstract (arXiv)

We demonstrate that Principal Component Analysis (PCA), when applied in a structured manner, either to polar-transformed images or segment-wise to token sequences, enables extreme compression of neural models without sacrificing performance. Across three case studies, we show that a one-layer classifier trained on PCA-compressed polar MNIST achieves over 98 percent accuracy using only 840 parameters. A two-layer transformer trained on 70-dimensional PCA-reduced MiniLM embeddings reaches 76.62 percent accuracy on the 20 Newsgroups dataset with just 81000 parameters. A decoder-only transformer generates coherent token sequences from 70-dimensional PCA embeddings while preserving over 97 percent cosine similarity with full MiniLM representations, using less than 17 percent of the parameter count of GPT-2. These results highlight PCA-based input compression as a general and effective strategy for aligning model capacity with information content, enabling lightweight architectures across multiple modalities.