mlsanityData to 7 Oct 2026

MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation

arXiv 2108.08447 ↗First posted 19 Aug 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Conditional masked language models (CMLM) have shown impressive progress in non-autoregressive machine translation (NAT). They learn the conditional translation model by predicting the random masked subset in the target sentence. Based on the CMLM framework, we introduce Multi-view Subset Regularization (MvSR), a novel regularization method to improve the performance of the NAT model. Specifically, MvSR consists of two parts: (1) textitshared mask consistency: we forward the same target with different mask strategies, and encourage the predictions of shared mask positions to be consistent with each other. (2) textitmodel consistency, we maintain an exponential moving average of the model weights, and enforce the predictions to be consistent between the average model and the online model. Without changing the CMLM-based architecture, our approach achieves remarkable performance on three public benchmarks with 0.36-1.14 BLEU gains over previous NAT models. Moreover, compared with the stronger Transformer baseline, we reduce the gap to 0.01-0.44 BLEU scores on small datasets (WMT16 ROleftrightarrowEN and IWSLT DErightarrowEN).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

42 of 51 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
30 more
  1. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2017Deep Mutual Learningcited by 621
  5. 2017Convolutional Sequence to Sequence Learningcited by 550
  6. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  7. 2017Non-Autoregressive Neural Machine Translationcited by 457
  8. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  9. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  10. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  13. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  14. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  15. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  16. 2019Levenshtein Transformercited by 197
  17. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  18. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  19. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  20. 2020Adversarial Training for Large Neural Language Modelscited by 109
  21. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  22. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  23. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  24. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  25. 2019Fast Structured Decoding for Sequence Modelscited by 59
  26. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  27. 2019Imitation Learning for Non-Autoregressive Neural Machine Translationcited by 36
  28. 2020Semi-Autoregressive Training Improves Mask-Predict Decodingcited by 33
  29. 2019Hint-Based Training for Non-Autoregressive Machine Translationcited by 28
  30. 2020Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanismcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 2.

5 papers.

  1. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  2. 2023AMOM: Adaptive Masking over Masking for Conditional Masked Language Modelcited by 5
  3. 2023N-Gram Nearest Neighbor Machine Translationcited by 1
  4. 2024Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularizationcited by 0
  5. 2024Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translationcited by 0