mlsanityData to 7 Oct 2026

Fast Vocabulary Transfer for Language Model Compression

arXiv 2402.09977 ↗First posted 15 Feb 2024cs.CL cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Real-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertical domains and downstream tasks. Our results indicate that vocabulary transfer can be effectively used in combination with other compression techniques, yielding a significant reduction in model size and inference time while marginally compromising on performance.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

20 of 24 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
8 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  3. 2015Deep Learning with Limited Numerical Precisioncited by 368
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  6. 2018Model compression via distillation and quantizationcited by 287
  7. 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14
  8. 2020Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 16; 2026: 11.

42 papers.

Sort:

By area:

  1. 2025SuperBPE: Space Travel for Language Modelscited by 58
  2. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  3. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  4. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  5. 2024Zero-Shot Tokenizer Transfercited by 30
  6. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  7. 2024Exploring Design Choices for Building Language-Specific LLMscited by 22
  8. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  9. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  10. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  11. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  12. 2025Sampling from Your Language Model One Byte at a Timecited by 7
30 more
  1. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
  2. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  3. 2025A Survey of Classification Tasks and Approaches for Legal Contractscited by 6
  4. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  5. 2024ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Modelcited by 5
  6. 2024Generation with Dynamic Vocabularycited by 5
  7. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  8. 2025Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuitcited by 4
  9. 2024Are Compressed Language Models Less Subgroup Robust?cited by 3
  10. 2025Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estoniancited by 3
  11. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  12. 2026AdaptBPE: From General Purpose to Specialized Tokenizerscited by 3
  13. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformerscited by 2
  14. 2025Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptationcited by 2
  15. 2025Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfercited by 2
  16. 2025Lossless Vocabulary Reduction for Auto-Regressive Language Modelscited by 2
  17. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  18. 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
  19. 2024Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflowscited by 1
  20. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 1
  21. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  22. 2026TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignmentcited by 1
  23. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translationcited by 0
  24. 2026Texo: Formula Recognition within 20M Parameterscited by 0
  25. 2026Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Codecited by 0
  26. 2026Echoes Across Vietnam's Highlands, Delta, and Coast: A Multilingual Corpus for Cham, Khmer, and Tay-Nungcited by 0
  27. 2026Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extensioncited by 0
  28. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decodingcited by 0
  29. 2026BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Modelscited by 0
  30. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0