mlsanityData to 7 Oct 2026

Neural Machine Translation with Byte-Level Subwords

arXiv 1909.03341 ↗First posted 7 Sep 2019cs.CLcited by 115

Not on the map. Why?

Abstract (arXiv)

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 16; 2023: 17; 2024: 35; 2025: 25; 2026: 8.

115 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  3. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  4. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  5. 2021Direct speech-to-speech translation with discrete unitscited by 148
  6. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  7. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  8. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  9. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  10. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  11. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  12. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
38 more
  1. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  2. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  3. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  4. 2022Language Modelling with Pixelscited by 53
  5. 2021A Survey on Green Deep Learningcited by 52
  6. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  7. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  8. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  9. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  10. 2023A Comparative Study of Pretrained Language Models for Long Clinical Textcited by 35
  11. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  12. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  13. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  14. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  15. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  16. 2023Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generationcited by 24
  17. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 23
  18. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  19. 2024Beyond Language Models: Byte Models are Digital World Simulatorscited by 21
  20. 2024EXAONE 3.0 7.8B Instruction Tuned Language Modelcited by 20
  21. 2021Zero-Shot Cross-lingual Semantic Parsingcited by 19
  22. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  23. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  24. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  25. 2023A Sequence-to-Sequence&Set Model for Text-to-Table Generationcited by 16
  26. 2024The Hidden Space of Transformer Language Adapterscited by 16
  27. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  28. 2023Benchmarking Causal Study to Interpret Large Language Models for Source Codecited by 14
  29. 2023PersianLLaMA: Towards Building First Persian Large Language Modelcited by 14
  30. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  31. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 13
  32. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  33. 2024Tele-FLM Technical Reportcited by 12
  34. 2022Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRscited by 11
  35. 2025Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledgecited by 10
  36. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  37. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  38. 2025See the Text: From Tokenization to Visual Readingcited by 9

50 most cited of 115 All 115 (JSON)