mlsanityData to 14 Oct 2026

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

arXiv 1804.10959 ↗First posted 29 Apr 2018cs.CLcited by 639

Not on the map. Why?

Abstract (arXiv)

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

Key facts. Cited by 639 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 104; 2022: 93; 2023: 115; 2024: 137; 2025: 97; 2026: 93.

639 papers.

Sort:

By area:

  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  3. 2023BloombergGPT: A Large Language Model for Financecited by 939
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  8. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  9. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  10. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  11. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  12. 2021Few-shot Learning with Multilingual Language Modelscited by 278
38 more
  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  2. 2023Language Modeling Is Compressioncited by 225
  3. 2022Continuous diffusion for categorical datacited by 214
  4. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  5. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  7. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  8. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  9. 2021Direct speech-to-speech translation with discrete unitscited by 149
  10. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  11. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  12. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  13. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  14. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  15. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  16. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  17. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  18. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  19. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  20. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  21. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  22. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  23. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  24. 2021Survey of Low-Resource Machine Translationcited by 92
  25. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  26. 2024Tokenization Is More Than Compressioncited by 89
  27. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  28. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  29. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  30. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  31. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  32. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  33. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  34. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  35. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  36. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  37. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 66
  38. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63

50 most cited of 639 All 639 (JSON)