mlsanityData to 7 Oct 2026

Token Distillation: Attention-aware Input Embeddings For New Tokens

arXiv 2505.20133 ↗First posted 26 May 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

52 of 66 references.

References · 52

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  6. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  10. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  11. 20242 OLMo 2 Furiouscited by 302
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
38 more
  1. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  2. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  3. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  4. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  5. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  6. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  7. 2025SuperBPE: Space Travel for Language Modelscited by 58
  8. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  9. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  10. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  11. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  12. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  13. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  14. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  15. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  16. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  17. 2017Mimicking Word Embeddings using Subword RNNscited by 30
  18. 2024Zero-Shot Tokenizer Transfercited by 30
  19. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  20. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  21. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  22. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  23. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  24. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23
  25. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  26. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  27. 2019BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performancecited by 18
  28. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  29. 2019Attentive Mimicking: Better Word Embeddings by Attending to Informative Contextscited by 17
  30. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  31. 2024Multi-word Tokenization for Sequence Compressioncited by 16
  32. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  33. 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14
  34. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  35. 2017One-shot and few-shot learning of word embeddingscited by 7
  36. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  37. 2024CoLLEGe: Concept Embedding Generation for Large Language Modelscited by 3
  38. 2022Vocabulary Transfer for Biomedical Texts: Add Tokens if You Can Not Add Datacited by 2

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  2. 2025Model-Aware Tokenizer Transfercited by 5
  3. 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
  4. 2026Laguna M.1/XS.2 Technical Reportcited by 1
  5. 2026In-Place Tokenizer Expansion for Pre-trained LLMscited by 0
  6. 2026Disentangling Language Modeling and Boundariescited by 0