mlsanityData to 7 Oct 2026

Direction is what you need: Improving Word Embedding Compression in Large Language Models

arXiv 2106.08181 ↗First posted 15 Jun 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of these models to improve their inference time and memory footprint. This paper presents a novel loss objective to compress token embeddings in the Transformer-based models by leveraging an AutoEncoder architecture. More specifically, we emphasize the importance of the direction of compressed embeddings with respect to original uncompressed embeddings. The proposed method is task-agnostic and does not require further language modeling pre-training. Our method significantly outperforms the commonly used SVD-based matrix-factorization approach in terms of initial language model Perplexity. Moreover, we evaluate our proposed approach over SQuAD v1.1 dataset and several downstream tasks from the GLUE benchmark, where we also outperform the baseline in most scenarios. Our code is public.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
11 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  7. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  8. 2019Structured Pruning of Large Language Modelscited by 201
  9. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  10. 2019On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compressioncited by 14
  11. 2019Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decompositioncited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 3.

4 papers.

  1. 2023Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Modelscited by 1
  2. 2025Lyrics Matter: Exploiting the Power of Learnt Representations for Music Popularity Predictioncited by 1
  3. 2025TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devicescited by 0
  4. 2025Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transformcited by 0