mlsanityData to 14 Oct 2026

AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain

arXiv 2110.13434 ↗First posted 26 Oct 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain-specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 12 of its 19 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

12 of 19 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  9. 2015Image-based Recommendations on Styles and Substitutescited by 706
  10. 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
  11. 2019Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimickingcited by 41
  12. 2020Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QAcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 7; 2025: 4; 2026: 3.

20 papers.

Sort:

By area:

  1. 2022Language Modelling with Pixelscited by 55
  2. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  3. 2024Accelerating Multilingual Language Model for Excessively Tokenized Languagescited by 8
  4. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  5. 2024Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMscited by 7
  6. 2024The Impact of Word Splitting on the Semantic Content of Contextualized Word Representationscited by 6
  7. 2023RadLing: Towards Efficient Radiology Report Understandingcited by 5
  8. 2025ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddingscited by 5
  9. 2023Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Modelscited by 4
  10. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  11. 2022Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gapscited by 2
  12. 2024Infusing clinical knowledge into tokenisers for language modelscited by 2
8 more
  1. 2024Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Modelscited by 1
  2. 2024Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflowscited by 1
  3. 2023Painsight: An Extendable Opinion Mining Framework for Detecting Pain Points Based on Online Customer Reviewscited by 0
  4. 2025ManufactuBERT: Efficient Continual Pretraining for Manufacturingcited by 0
  5. 2025Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extractioncited by 0
  6. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  7. 2026Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Codecited by 0
  8. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0