mlsanityData to 14 Oct 2026

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models

arXiv 2306.08756 ↗First posted 14 Jun 2023cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency by initializing one model from the other. (1) Extracting the encoder from a seq2seq model, we show it under-performs a Masked Language Modeling (MLM) encoder, particularly on sequence labeling tasks. Variations of masking during seq2seq training, reducing the decoder size, and continuing with a small amount of MLM training do not close the gap. (2) Conversely, using an encoder to warm-start seq2seq training, we show that by unfreezing the encoder partway through training, we can match task performance of a from-scratch seq2seq model. Overall, this two-stage approach is an efficient recipe to obtain both a multilingual encoder and a seq2seq model, matching the performance of training each model from scratch while reducing the total compute cost by 27%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
20 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  7. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  8. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  9. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  10. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  11. 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
  12. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  13. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  14. 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 83
  15. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  16. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  17. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
  18. 2022CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsingcited by 21
  19. 2022Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systemscited by 15
  20. 2021EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Modelscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Modelscited by 1