mlsanityData to 7 Oct 2026

CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

arXiv 2308.00683 ↗First posted 1 Aug 2023cs.LG cs.CL cs.SEcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

36 of 48 references.

References · 36

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
24 more
  1. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  2. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  3. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  4. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  5. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2020Unsupervised Translation of Programming Languagescited by 262
  8. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  9. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  10. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  11. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  12. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  13. 2018Tensor2Tensor for Neural Machine Translationcited by 85
  14. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  15. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  16. 2020Big Code != Big Vocabulary: Open-Vocabulary Models for Source Codecited by 53
  17. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  18. 2021Multi-view Subword Regularizationcited by 40
  19. 2019Program Synthesis and Semantic Parsing with Learned Code Idiomscited by 34
  20. 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
  21. 2019Learning Programmatic Idioms for Scalable Semantic Parsingcited by 17
  22. 2019Character-based NMT with Transformercited by 10
  23. 2021Efficient Inference for Multilingual Neural Machine Translationcited by 10
  24. 2020A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Codecited by 6

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3; 2025: 3.

9 papers.

  1. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  2. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  3. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  4. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  5. 2024AI Coders Are Among Us: Rethinking Programming Language Grammar Towards Efficient Code Generationcited by 8
  6. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  7. 2023Learn Your Tokens: Word-Pooled Tokenization for Language Modelingcited by 7
  8. 2025TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammarcited by 2
  9. 2024FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproductioncited by 1