mlsanityData to 7 Oct 2026

Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling

arXiv 2501.16975 ↗First posted 28 Jan 2025cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Tokenization is a fundamental component of large language models (LLMs), yet its influence on model scaling and performance is not fully explored. In this paper, we introduce Over-Tokenized Transformers, a novel framework that decouples input and output vocabularies to improve language modeling performance. Specifically, our approach scales up input vocabularies to leverage multi-gram tokens. Through extensive experiments, we uncover a log-linear relationship between input vocabulary size and training loss, demonstrating that larger input vocabularies consistently enhance model performance, regardless of model size. Using a large input vocabulary, we achieve performance comparable to double-sized baselines with no additional cost. Our findings highlight the importance of tokenization in scaling laws and provide practical insight for tokenizer design, paving the way for more efficient and powerful LLMs.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

24 of 26 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  12. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
12 more
  1. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  4. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  5. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  6. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  7. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  8. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  9. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  10. 2022N-Grammer: Augmenting Transformers with latent n-gramscited by 13
  11. 2021Lookup-Table Recurrent Language Models for Long Tail Speech Recognitioncited by 9
  12. 2022A Frequency-aware Software Cache for Large Recommendation System Embeddingscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 24; 2026: 18.

42 papers.

Sort:

By area:

  1. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  2. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  3. 2025SuperBPE: Space Travel for Language Modelscited by 58
  4. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  5. 2025Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barriercited by 31
  6. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  7. 2025Scaling Embedding Layers in Language Modelscited by 19
  8. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  9. 2026Scaling Embeddings Outperforms Scaling Experts in Language Modelscited by 13
  10. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  11. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  12. 2025Frac-Connections: Fractional Extension of Hyper-Connectionscited by 8
30 more
  1. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  2. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  3. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  4. 2026Speculative Decoding with a Speculative Vocabularycited by 6
  5. 2025XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decodingcited by 5
  6. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  7. 2026Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Modelscited by 5
  8. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  9. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  10. 2025Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Modelscited by 3
  11. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  12. 2025Virtual Width Networkscited by 3
  13. 2026Proxy Compression for Language Modelingcited by 3
  14. 2026LiteToken: Removing Intermediate Merge Residues From BPE Tokenizerscited by 3
  15. 2026Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenizationcited by 3
  16. 2025SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performancecited by 2
  17. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  18. 2026Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulationcited by 2
  19. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 1
  20. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  21. 2025Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Modelscited by 1
  22. 2026Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scalingcited by 1
  23. 2026Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMscited by 1
  24. 2025Mixture of Lookup Key-Value Expertscited by 0
  25. 2026Training Language Models with homotokens Leads to Delayed Overfittingcited by 0
  26. 2026Investigating Gender Stereotypes in Large Language Models via Social Determinants of Healthcited by 0
  27. 2026Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspectivecited by 0
  28. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
  29. 2026Softmax Reparameterization for Output-Head Quantizationcited by 0
  30. 2026Forking: Sudden Overfitting Under Replaycited by 0