mlsanityData to 7 Oct 2026

Tensorized Embedding Layers for Efficient Model Compression

arXiv 1901.10787 ↗First posted 30 Jan 2019cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 5; 2024: 6; 2025: 1; 2026: 2.

27 papers.

Sort:

  1. 2021TT-Rec: Tensor Train Compression for Deep Learning Recommendation Modelscited by 58
  2. 2021Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systemscited by 29
  3. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  4. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  5. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  6. 2023Tensor Decomposition for Model Reduction in Neural Networks: A Reviewcited by 14
  7. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 14
  8. 2024Motion meets Attention: Video Motion Promptscited by 12
  9. 2021Binary Code based Hash Embedding for Web-scale Applicationscited by 11
  10. 2023Efficient GPT Model Pre-training using Tensor Train Matrix Representationcited by 8
  11. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
  12. 2022tntorch: Tensor Network Learning with PyTorchcited by 7
15 more
  1. 2024Super Tiny Language Modelscited by 7
  2. 2022Nimble GNN Embedding with Tensor-Train Decompositioncited by 6
  3. 2024TQCompressor: improving tensor decomposition methods in neural networks via permutationscited by 5
  4. 2024A Matrix Product State Model for Simultaneous Classification and Generationcited by 5
  5. 20213U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Accelerationcited by 3
  6. 2023Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problemcited by 3
  7. 2024Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Modelscited by 3
  8. 2021From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tablescited by 2
  9. 2022Deep tensor networks with matrix product operatorscited by 2
  10. 2026Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPTcited by 2
  11. 2021Learning Robust and Lightweight Model through Separable Structured Transformationscited by 0
  12. 2023Review of compressed embedding layers and their applications for recommender systemscited by 0
  13. 2023Enhancing Cross-Category Learning in Recommendation Systems with Multi-Layer Embedding Trainingcited by 0
  14. 2025SCRec: A Scalable Computational Storage System with Statistical Sharding and Tensor-train Decomposition for Recommendation Modelscited by 0
  15. 2026Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scalecited by 0