mlsanityData to 7 Oct 2026

Multi-view Subword Regularization

arXiv 2103.08490 ↗First posted 15 Mar 2021cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary. However, standard heuristic algorithms often lead to sub-optimal segmentation, especially for languages with limited amounts of data. In this paper, we take two major steps towards alleviating this problem. First, we demonstrate empirically that applying existing subword regularization methods(Kudo, 2018; Provilkov et al., 2020) during fine-tuning of pre-trained multilingual representations improves the effectiveness of cross-lingual transfer. Second, to take full advantage of different possible input segmentations, we propose Multi-view Subword Regularization (MVR), a method that enforces the consistency between predictions of using inputs tokenized by the standard and probabilistic segmentations. Results on the XTREME multilingual benchmark(Hu et al., 2020) show that MVR brings consistent improvements of up to 2.5 points over using standard segmentation algorithms.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 36 of its 39 references that are in the map. First posted on arXiv 15 Mar 2021.

Builds on

36 of 39 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
24 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  6. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  7. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  8. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  9. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  10. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  11. 2016Exploring the Limits of Language Modelingcited by 172
  12. 2015Character-Aware Neural Language Modelscited by 129
  13. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  14. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  15. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  16. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  17. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  18. 2020Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translationcited by 38
  19. 2019Revisiting Knowledge Distillation via Label Smoothing Regularizationcited by 29
  20. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  21. 2020AMBERT: A Pre-trained Language Model with Multi-Grained Tokenizationcited by 22
  22. 2019Multilingual Neural Machine Translation With Soft Decoupled Encodingcited by 21
  23. 2018Compositional Representation of Morphologically-Rich Input for Neural Machine Translationcited by 12
  24. 2017Cross-lingual Distillation for Text Classificationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 6; 2024: 7; 2025: 3; 2026: 5.

40 papers.

Sort:

By area:

  1. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
  2. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  3. 2022Language Modelling with Pixelscited by 53
  4. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  5. 2022The SIGMORPHON 2022 Shared Task on Morpheme Segmentationcited by 25
  6. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  7. 2022Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languagescited by 20
  8. 2021Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noisecited by 19
  9. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  10. 2021Evaluating Various Tokenizers for Arabic Text Classificationcited by 14
  11. 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14
  12. 2022Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimizationcited by 13
28 more
  1. 2021Subword Mapping and Anchoring across Languagescited by 12
  2. 2022A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learningcited by 12
  3. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  4. 2021Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?cited by 11
  5. 2022CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translationcited by 9
  6. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 9
  7. 2022Subword Segmental Language Modelling for Nguni Languagescited by 8
  8. 2023Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Studycited by 8
  9. 2021How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?cited by 7
  10. 2023Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generationcited by 7
  11. 2023Learn Your Tokens: Word-Pooled Tokenization for Language Modelingcited by 7
  12. 2022Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modelingcited by 6
  13. 2023BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Textcited by 6
  14. 2022Local Byte Fusion for Neural Machine Translationcited by 5
  15. 2025Speculating LLMs' Chinese Training Data Pollution from Their Tokenscited by 5
  16. 2024Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generationcited by 3
  17. 2024A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translationcited by 3
  18. 2025SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performancecited by 2
  19. 2025The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languagescited by 2
  20. 2024SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularizationcited by 1
  21. 2026Consistency Training Can Entrench Misalignmentcited by 1
  22. 2023Elementwise Language Representationcited by 0
  23. 2024Generalization Measures for Zero-Shot Cross-Lingual Transfercited by 0
  24. 2024Deterministic Reversible Data Augmentation for Neural Machine Translationcited by 0
  25. 2026TREX: Tokenizer Regression for Optimal Data Mixturecited by 0
  26. 2026Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLPcited by 0
  27. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0
  28. 2026TOAST: Stochastic Robot Action Tokenization for Autoregressive Vision-Language-Action Modelscited by 0