# AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain

arXiv: https://arxiv.org/abs/2110.13434 · HTML: https://mlsanity.com/paper/2110.13434/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://mlsanity.com/paper/1410.8206/) (2014)
- [Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking](https://mlsanity.com/paper/1904.06707/) (2019)
- [Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA](https://mlsanity.com/paper/2004.03354/) (2020)

## Built on by (most cited first, 20 of 20)

- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining](https://mlsanity.com/paper/2311.08849/) (2023)
- [Accelerating Multilingual Language Model for Excessively Tokenized Languages](https://mlsanity.com/paper/2401.10660/) (2024)
- [MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization](https://mlsanity.com/paper/2405.04163/) (2024)
- [Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs](https://mlsanity.com/paper/2403.05434/) (2024)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [RadLing: Towards Efficient Radiology Report Understanding](https://mlsanity.com/paper/2306.02492/) (2023)
- [ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings](https://mlsanity.com/paper/2508.01643/) (2025)
- [Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models](https://mlsanity.com/paper/2302.07912/) (2023)
- [Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings](https://mlsanity.com/paper/2505.21242/) (2025)
- [Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps](https://mlsanity.com/paper/2211.03988/) (2022)
- [Infusing clinical knowledge into tokenisers for language models](https://mlsanity.com/paper/2406.14312/) (2024)
- [Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models](https://mlsanity.com/paper/2410.03258/) (2024)
- [Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows](https://mlsanity.com/paper/2411.19295/) (2024)
- [Painsight: An Extendable Opinion Mining Framework for Detecting Pain Points Based on Online Customer Reviews](https://mlsanity.com/paper/2306.02043/) (2023)
- [ManufactuBERT: Efficient Continual Pretraining for Manufacturing](https://mlsanity.com/paper/2511.05135/) (2025)
- [Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://mlsanity.com/paper/2512.11502/) (2025)
- [Task-Centric Acceleration of Small-Language Models](https://mlsanity.com/paper/2602.24174/) (2026)
- [Supporting Workflow Reproducibility by Linking Bioinformatics Tools across Papers and Executable Code](https://mlsanity.com/paper/2603.08195/) (2026)
- [Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization](https://mlsanity.com/paper/2605.17379/) (2026)

## Abstract (arXiv)

During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain-specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).