mlsanityData to 7 Oct 2026

Multilingual Translation via Grafting Pre-trained Language Models

arXiv 2109.05256 ↗First posted 11 Sep 2021cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pre-trained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 11 Sep 2021.

Builds on

45 of 58 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
33 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  9. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  10. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  13. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  14. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  15. 2019Massively Multilingual Neural Machine Translationcited by 246
  16. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  17. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  18. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  19. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  20. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  21. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  22. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  23. 2019Multilingual Neural Machine Translation with Language Clusteringcited by 47
  24. 2016Zero-Resource Translation with Multi-Lingual Neural Machine Translationcited by 46
  25. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  26. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  27. 2020Code-switching pre-training for neural machine translationcited by 43
  28. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  29. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  30. 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
  31. 2018Multilingual Neural Machine Translation with Task-Specific Attentioncited by 26
  32. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24
  33. 2019Multilingual Neural Machine Translation With Soft Decoupled Encodingcited by 21

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 3.

10 papers.

  1. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  2. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  3. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  4. 2023A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directionscited by 22
  5. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  6. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  7. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  8. 2022Zero-shot Domain Adaptation for Neural Machine Translation with Retrieved Phrase-level Promptscited by 7
  9. 2023E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translationcited by 5
  10. 2022Better Datastore, Better Translation: Generating Datastores from Pre-Trained Models for Nearest Neural Machine Translationcited by 2