# Tensorized Embedding Layers for Efficient Model Compression

arXiv: https://arxiv.org/abs/1901.10787 · HTML: https://mlsanity.com/paper/1901.10787/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2019.

## Built on by (most cited first, 27 of 27)

- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization](https://mlsanity.com/paper/2405.14377/) (2024)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Motion meets Attention: Video Motion Prompts](https://mlsanity.com/paper/2407.03179/) (2024)
- [Binary Code based Hash Embedding for Web-scale Applications](https://mlsanity.com/paper/2109.02471/) (2021)
- [Efficient GPT Model Pre-training using Tensor Train Matrix Representation](https://mlsanity.com/paper/2306.02697/) (2023)
- [Knowledge Distillation of Russian Language Models with Reduction of Vocabulary](https://mlsanity.com/paper/2205.02340/) (2022)
- [tntorch: Tensor Network Learning with PyTorch](https://mlsanity.com/paper/2206.11128/) (2022)
- [Super Tiny Language Models](https://mlsanity.com/paper/2405.14159/) (2024)
- [Nimble GNN Embedding with Tensor-Train Decomposition](https://mlsanity.com/paper/2206.10581/) (2022)
- [A Matrix Product State Model for Simultaneous Classification and Generation](https://mlsanity.com/paper/2406.17441/) (2024)
- [TQCompressor: improving tensor decomposition methods in neural networks via permutations](https://mlsanity.com/paper/2401.16367/) (2024)
- [Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models](https://mlsanity.com/paper/2410.03040/) (2024)
- [3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration](https://mlsanity.com/paper/2105.06250/) (2021)
- [Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem](https://mlsanity.com/paper/2301.08441/) (2023)
- [Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT](https://mlsanity.com/paper/2603.28534/) (2026)
- [From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables](https://mlsanity.com/paper/2104.08677/) (2021)
- [Deep tensor networks with matrix product operators](https://mlsanity.com/paper/2209.09098/) (2022)
- [Learning Robust and Lightweight Model through Separable Structured Transformations](https://mlsanity.com/paper/2112.13551/) (2021)
- [Review of compressed embedding layers and their applications for recommender systems](https://mlsanity.com/paper/2306.13724/) (2023)
- [Enhancing Cross-Category Learning in Recommendation Systems with Multi-Layer Embedding Training](https://mlsanity.com/paper/2309.15881/) (2023)
- [SCRec: A Scalable Computational Storage System with Statistical Sharding and Tensor-train Decomposition for Recommendation Models](https://mlsanity.com/paper/2504.00520/) (2025)
- [Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale](https://mlsanity.com/paper/2609.34292/) (2026)

## Abstract (arXiv)

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.