mlsanityData to 14 Oct 2026

Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation

arXiv 2206.02368 ↗First posted 6 Jun 2022cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce Bi-SimCut: a simple but effective training strategy to boost neural machine translation (NMT) performance. It consists of two procedures: bidirectional pretraining and unidirectional finetuning. Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sentence pairs. Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31.16 for en -> de and 38.37 for de -> en on the IWSLT14 dataset, 30.78 for en -> de and 35.15 for de -> en on the WMT14 dataset, and 27.17 for zh -> en on the WMT17 dataset. SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the universality and simplicity of SimCut and Bi-SimCut, we believe they can serve as strong baselines for future NMT research.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

26 of 36 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  11. 2020Overfitting in adversarially robust deep learningcited by 506
  12. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
14 more
  1. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  2. 2014Learning with Pseudo-Ensemblescited by 241
  3. 2018Scaling Neural Machine Translationcited by 204
  4. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  5. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  6. 2019The Evolved Transformercited by 180
  7. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  8. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 106
  9. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  10. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  11. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  12. 2021Rethinking Perturbations in Encoder-Decoders for Fast Trainingcited by 17
  13. 2021Improving Neural Machine Translation by Bidirectional Trainingcited by 17
  14. 2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 11

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 6; 2024: 1.

8 papers.

  1. 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
  2. 2023Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularizationcited by 10
  3. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  4. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  5. 2022ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translationcited by 4
  6. 2023VBD-MT Chinese-Vietnamese Translation Systems for VLSP 2022cited by 1
  7. 2023Integrating Pre-trained Language Model into Neural Machine Translationcited by 0
  8. 2023Unraveling Key Factors of Knowledge Distillationcited by 0