mlsanityData to 7 Oct 2026

Fine-Tuning Transformers: Vocabulary Transfer

arXiv 2112.14569 ↗First posted 29 Dec 2021cs.CL cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Transformers are responsible for the vast majority of recent advances in natural language processing. The majority of practical natural language processing applications of these models are typically enabled through transfer learning. This paper studies if corpus-specific tokenization used for fine-tuning improves the resulting performance of the model. Through a series of experiments, we demonstrate that such tokenization combined with the initialization and fine-tuning strategy for the vocabulary tokens speeds up the transfer and boosts the performance of the fine-tuned model. We call this aspect of transfer facilitation vocabulary transfer.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 29 Dec 2021.

Builds on

14 of 24 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  7. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  8. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  9. 2021Multi-view Subword Regularizationcited by 40
  10. 2019Controlling the Output Length of Neural Machine Translationcited by 36
  11. 2020MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretrainingcited by 18
  12. 2019Transfer Fine-Tuning: A BERT Case Studycited by 8
2 more
  1. 2019Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignmentcited by 3
  2. 2020The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Taskcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 7; 2025: 3; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  2. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  3. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  4. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  5. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  6. 2022BERT in Plutarch's Shadowscited by 6
  7. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  8. 2024Towards a RAG-based Summarization Agent for the Electron-Ion Collidercited by 5
  9. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 4
  10. 2024TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domaincited by 3
  11. 2024Infusing clinical knowledge into tokenisers for language modelscited by 2
  12. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
2 more
  1. 2026Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructurecited by 1
  2. 2026Texo: Formula Recognition within 20M Parameterscited by 0