# On the Downstream Performance of Compressed Word Embeddings

arXiv: https://arxiv.org/abs/1909.01264 · HTML: https://mlsanity.com/paper/1909.01264/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 13 of 13)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features](https://mlsanity.com/paper/2302.05441/) (2023)
- [SDR: Efficient Neural Re-ranking using Succinct Document Representation](https://mlsanity.com/paper/2110.02065/) (2021)
- [Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems](https://mlsanity.com/paper/2108.05053/) (2021)
- [BERTops: Studying BERT Representations under a Topological Lens](https://mlsanity.com/paper/2205.00953/) (2022)
- [Look-ups are not (yet) all you need for deep learning inference](https://mlsanity.com/paper/2207.05808/) (2022)
- [On Geodesic Distances and Contextual Embedding Compression for Text Classification](https://mlsanity.com/paper/2104.11295/) (2021)
- [RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression](https://mlsanity.com/paper/2605.14359/) (2026)
- [Compressing Large Language Models with PCA Without Performance Loss](https://mlsanity.com/paper/2508.04307/) (2025)
- [Normalized Space Alignment: A Versatile Metric for Representation Analysis](https://mlsanity.com/paper/2411.04512/) (2024)
- [From Layers to Networks: Comparing Neural Representations via Diffusion Geometry](https://mlsanity.com/paper/2605.15901/) (2026)
- [Subdomain-aware representation compression for pretrained image embeddings](https://mlsanity.com/paper/2609.20213/) (2026)

## Abstract (arXiv)

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to 2× lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.