mlsanityData to 14 Oct 2026

AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation

arXiv 2503.19693 ↗First posted 25 Mar 2025cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive versatility as general purpose models. However, their broad applicability comes at a high-cost computational overhead, particularly in auto-regressive decoding where each step requires a forward pass. In domain-specific settings, general-purpose capabilities are unnecessary and can be exchanged for efficiency. In this work, we take a novel perspective on domain adaptation, reducing latency and computational costs by adapting the vocabulary to focused domains of interest. We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation, designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be applied to any tokenizer and architecture, modifying the vocabulary by replacing tokens with domain-specific n-gram-based tokens, thereby reducing the number of tokens required for both input processing and output generation. AdaptiVocab initializes new n-token embeddings using an exponentially weighted combination of existing embeddings and employs a lightweight fine-tuning phase that can be efficiently performed on a single GPU. We evaluate two 7B LLMs across three niche domains, assessing efficiency, generation quality, and end-task performance. Our results show that AdaptiVocab reduces token usage by over 25% without compromising performance

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 60 of its 76 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

60 of 76 references.

References · 60

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  6. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  7. 2019Green AIcited by 627
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  9. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  11. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  12. 2016Using the Output Embedding to Improve Language Modelscited by 272
38 more
  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  2. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  4. 2023Efficient Large Language Models: A Surveycited by 186
  5. 2025Qwen2.5-1M Technical Reportcited by 164
  6. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  7. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  8. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  9. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  10. 2024Tokenization Is More Than Compressioncited by 89
  11. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  12. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  13. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  14. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
  15. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  16. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  17. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  18. 2025SuperBPE: Space Travel for Language Modelscited by 63
  19. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  20. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  21. 2023Tokenization and the Noiseless Channelcited by 59
  22. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  23. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 37
  24. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  25. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 36
  26. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  27. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  28. 2024Zero-Shot Tokenizer Transfercited by 31
  29. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  30. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  31. 2023MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Promptingcited by 27
  32. 2022M2D2: A Massively Multi-domain Language Modeling Datasetcited by 26
  33. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 23
  34. 2021Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapterscited by 21
  35. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  36. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  37. 2024A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Lengthcited by 16
  38. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  2. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  3. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  4. 2025Multi-Domain Explainability of Preferencescited by 2
  5. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  6. 2025Vocabulary Customization for Efficient Domain-Specific LLM Deploymentcited by 1
  7. 2025ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architecturescited by 1
  8. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  9. 2026MultiHashFormer: Hash-based Generative Language Modelscited by 1
  10. 2026In-Place Tokenizer Expansion for Pre-trained LLMscited by 1
  11. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  12. 2026CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbationcited by 0
3 more
  1. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0
  2. 2026Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extensioncited by 0
  3. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0