mlsanityData to 7 Oct 2026

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

arXiv 1907.12461 ↗First posted 29 Jul 2019cs.CLcited by 182

Not on the map. Why?

Abstract (arXiv)

Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 51; 2022: 57; 2023: 40; 2024: 20; 2025: 8; 2026: 5.

182 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  4. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  5. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  6. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  7. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  8. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
  9. 2021A Simple Recipe for Multilingual Grammatical Error Correctioncited by 108
  10. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  11. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  12. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
38 more
  1. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  2. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  3. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  4. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  5. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  6. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  7. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  8. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  9. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  10. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  11. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  12. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  13. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  14. 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
  15. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  16. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  17. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  18. 2022EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Startcited by 35
  19. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  20. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  21. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  22. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  23. 2021Focus Attention: Promoting Faithfulness and Diversity in Summarizationcited by 30
  24. 2022Using Pre-Trained Language Models for Producing Counter Narratives Against Hate Speech: a Comparative Studycited by 30
  25. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  26. 2023Training-free Lexical Backdoor Attacks on Language Modelscited by 27
  27. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  28. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  29. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  30. 2022SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representationcited by 23
  31. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  32. 2021BASS: Boosting Abstractive Summarization with Unified Semantic Graphcited by 21
  33. 2021Sketch and Refine: Towards Faithful and Informative Table-to-Text Generationcited by 21
  34. 2021Structural Pre-training for Dialogue Comprehensioncited by 20
  35. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  36. 2021BiSECT: Learning to Split and Rephrase Sentences with Bitextscited by 18
  37. 2023Learning by Analogy: Diverse Questions Generation in Math Word Problemcited by 18
  38. 2022Assemble Foundation Models for Automatic Code Summarizationcited by 17

50 most cited of 182 All 182 (JSON)