# Denoising based Sequence-to-Sequence Pre-training for Text Generation

arXiv: https://arxiv.org/abs/1908.08206 · HTML: https://mlsanity.com/paper/1908.08206/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 15 of 15)

- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Unsupervised Neural Machine Translation with Generative Language Models Only](https://mlsanity.com/paper/2110.05448/) (2021)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting](https://mlsanity.com/paper/2101.00416/) (2021)
- [Ask2Transformers: Zero-Shot Domain labelling with Pre-trained Language Models](https://mlsanity.com/paper/2101.02661/) (2021)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Spelling Correction with Denoising Transformer](https://mlsanity.com/paper/2105.05977/) (2021)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [MAPGN: MAsked Pointer-Generator Network for sequence-to-sequence pre-training](https://mlsanity.com/paper/2102.07380/) (2021)
- [Linguistically-driven Multi-task Pre-training for Low-resource Neural Machine Translation](https://mlsanity.com/paper/2201.08070/) (2022)
- [Evaluation of Abstractive Summarisation Models with Machine Translation in Deliberative Processes](https://mlsanity.com/paper/2110.05847/) (2021)
- [When LLM Agents Meet Graph Optimization: An Automated Data Quality Improvement Approach](https://mlsanity.com/paper/2510.08952/) (2025)
- [OpenRTAG: A Comprehensive Benchmark for Robust Text-Attributed Graph Learning under Data Quality Degradation](https://mlsanity.com/paper/2607.19108/) (2026)

## Abstract (arXiv)

This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.