mlsanityData to 14 Oct 2026

Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs

arXiv 2410.01188 ↗First posted 2 Oct 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) demonstrate impressive generation abilities, they frequently struggle when it comes to specialized domains due to their limited domain-specific knowledge. Studies on domain-specific LLMs resort to expanding the vocabulary before fine-tuning on domain-specific corpus, aiming to decrease the sequence length and enhance efficiency during decoding, without thoroughly investigating the results of vocabulary expansion to LLMs over different domains. Our pilot study reveals that expansion with only a subset of the entire vocabulary may lead to superior performance. Guided by the discovery, this paper explores how to identify a vocabulary subset to achieve the optimal results. We introduce VEGAD, an adaptive method that automatically identifies valuable words from a given domain vocabulary. Our method has been validated through experiments on three Chinese datasets, demonstrating its effectiveness. Additionally, we have undertaken comprehensive analyses of the method. The selection of a optimal subset for expansion has shown to enhance performance on both domain-specific tasks and general tasks, showcasing the potential of VEGAD.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

17 of 26 references.

References · 17

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  6. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  7. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  8. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  9. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 131
  10. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  11. 2023FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasetscited by 69
  12. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 49
5 more
  1. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  2. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 23
  3. 2018Extending Pretrained Segmentation Networks with Additional Anatomical Structurescited by 22
  4. 2024More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMscited by 21
  5. 2021Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 6
  2. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
  3. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  4. 2025CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systemscited by 3
  5. 2026AdaptBPE: From General Purpose to Specialized Tokenizerscited by 3
  6. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  7. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  8. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  9. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  10. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0
  11. 2026Dual-Vocabulary Language Model for Cross-Tokenizer Distillationcited by 0
  12. 2026zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Losscited by 0