mlsanityData to 7 Oct 2026

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

arXiv 2101.00416 ↗First posted 2 Jan 2021cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective. SSR provides more fine-grained learning signals for text representations by supervising the model to rewrite imperfect spans to ground truth, and it is more consistent than text infilling with many downstream seq2seq tasks that rewrite a source sentences into a target sentence. Our experiments with T5 models on various seq2seq tasks show that SSR can substantially improve seq2seq pre-training. Moreover, we observe SSR is especially helpful to improve pre-training a small-size seq2seq model with a powerful imperfect span generator, which indicates a new perspective of transferring knowledge from a large model to a smaller model for seq2seq pre-training.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

45 of 67 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
33 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2016Sequence-Level Knowledge Distillationcited by 761
  13. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  14. 2019Defending Against Neural Fake Newscited by 553
  15. 2019Text Summarization with Pretrained Encoderscited by 553
  16. 2018Born Again Neural Networkscited by 539
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  18. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  19. 2017Toward Controlled Generation of Textcited by 271
  20. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  21. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  22. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  23. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  24. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  25. 2020Pre-training via Paraphrasingcited by 92
  26. 2019Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answeringcited by 63
  27. 2019An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correctioncited by 55
  28. 2018Harvesting Paragraph-Level Question-Answer Pairs from Wikipediacited by 49
  29. 2020Pre-training Text-to-Text Transformers for Concept-centric Common Sensecited by 42
  30. 2020Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correctioncited by 24
  31. 2019Improving Grammatical Error Correction with Machine Translation Pairscited by 17
  32. 2019Denoising based Sequence-to-Sequence Pre-training for Text Generationcited by 15
  33. 2020Pseudo-Bidirectional Decoding for Local Sequence Transductioncited by 2

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 7; 2023: 1; 2024: 1; 2025: 1.

13 papers.

Sort:

By area:

  1. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  4. 2021Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decodingcited by 23
  5. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  6. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  7. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  8. 2022A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Modelcited by 16
  9. 2022CodeEditor: Learning to Edit Source Code with Pre-trained Modelscited by 12
  10. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  11. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  12. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
1 more
  1. 2024Analyzing Sentiment Polarity Reduction in News Presentation through Contextual Perturbation and Large Language Modelscited by 1