mlsanityData to 7 Oct 2026

Large Vocabulary Size Improves Large Language Models

arXiv 2406.16508 ↗First posted 24 Jun 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary sizes lead to better performance in LLMs. Moreover, we consider a continual training scenario where a pre-trained language model is trained on a different target language. We introduce a simple method to use a new vocabulary instead of the pre-defined one. We show that using the new vocabulary outperforms the model with the vocabulary used in pre-training.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

34 of 48 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
22 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2020On Layer Normalization in the Transformer Architecturecited by 906
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  9. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  10. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  11. 2016Exploring the Limits of Language Modelingcited by 172
  12. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  13. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  14. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  15. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  16. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  17. 2016Efficient softmax approximation for GPUscited by 67
  18. 2021Why don't people use character-level machine translation?cited by 27
  19. 2022Cedille: A large autoregressive French language modelcited by 20
  20. 2019How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selectioncited by 11
  21. 2020All Word Embeddings from One Embeddingcited by 8
  22. 2019Character n-gram Embeddings to Improve RNN Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  2. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  3. 2025Model-Aware Tokenizer Transfercited by 5
  4. 2024The Fair Language Model Paradoxcited by 3
  5. 2025Hyperbolic Large Language Modelscited by 2
  6. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 2
  7. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  8. 2025Chem42: a Family of chemical Language Models for Target-aware Ligand Generationcited by 1
  9. 2026Barriers to Universal Reasoning With Transformers (And How to Overcome Them)cited by 1
  10. 2026NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabulariescited by 1
  11. 2026The Query Channel: Information-Theoretic Limits of Masking-Based Explanationscited by 0