# Online Embedding Compression for Text Classification using Low Rank Matrix Factorization

arXiv: https://arxiv.org/abs/1811.00641 · HTML: https://mlsanity.com/paper/1811.00641/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 22 of 22)

- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition](https://mlsanity.com/paper/2307.00526/) (2023)
- [Numerical Optimizations for Weighted Low-rank Estimation on Language Model](https://mlsanity.com/paper/2211.09718/) (2022)
- [Survey: Exploiting Data Redundancy for Optimization of Deep Learning](https://mlsanity.com/paper/2208.13363/) (2022)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [MorphTE: Injecting Morphology in Tensorized Embeddings](https://mlsanity.com/paper/2210.15379/) (2022)
- [Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks](https://mlsanity.com/paper/2306.00342/) (2023)
- [Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency](https://mlsanity.com/paper/2403.00625/) (2024)
- [Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores](https://mlsanity.com/paper/2408.11868/) (2024)
- [ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations](https://mlsanity.com/paper/2506.02818/) (2025)
- [Word Tour: One-dimensional Word Embeddings via the Traveling Salesman Problem](https://mlsanity.com/paper/2205.01954/) (2022)
- [Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications](https://mlsanity.com/paper/2405.15877/) (2024)
- [RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices](https://mlsanity.com/paper/2412.10856/) (2024)
- [IMPACT: Importance-Aware Activation Space Reconstruction](https://mlsanity.com/paper/2507.03828/) (2025)
- [Fast Exploration of Weight Sharing Opportunities for CNN Compression](https://mlsanity.com/paper/2102.01345/) (2021)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)
- [Smooth Model Compression without Fine-Tuning](https://mlsanity.com/paper/2505.24469/) (2025)
- [LFA applied to CNNs: Efficient Singular Value Decomposition of Convolutional Mappings by Local Fourier Analysis](https://mlsanity.com/paper/2506.05617/) (2025)
- [TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices](https://mlsanity.com/paper/2506.13514/) (2025)
- [Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform](https://mlsanity.com/paper/2508.00220/) (2025)
- [Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysis](https://mlsanity.com/paper/2604.23375/) (2026)
- [TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding](https://mlsanity.com/paper/2606.08184/) (2026)

## Abstract (arXiv)

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training,to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.