# Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

arXiv: https://arxiv.org/abs/2205.14521 · HTML: https://mlsanity.com/paper/2205.14521/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 28 May 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)
- [Learning Neural Templates for Text Generation](https://mlsanity.com/paper/1808.10122/) (2018)
- [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021)
- [BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle](https://mlsanity.com/paper/1909.07405/) (2019)
- [Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.08717/) (2019)
- [Unsupervised Paraphrasing by Simulated Annealing](https://mlsanity.com/paper/1909.03588/) (2019)
- [Unsupervised Text Generation by Learning from Search](https://mlsanity.com/paper/2007.08557/) (2020)
- [Non-Autoregressive Text Generation with Pre-trained Language Models](https://mlsanity.com/paper/2102.08220/) (2021)
- [Iterative Edit-Based Unsupervised Sentence Simplification](https://mlsanity.com/paper/2006.09639/) (2020)
- [BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining](https://mlsanity.com/paper/2012.15525/) (2020)
- [TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising](https://mlsanity.com/paper/2001.00725/) (2020)
- [Language as a Latent Variable: Discrete Generative Models for Sentence Compression](https://mlsanity.com/paper/1609.07317/) (2016)
- [SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression](https://mlsanity.com/paper/1904.03651/) (2019)
- [RefSum: Refactoring Neural Summarization](https://mlsanity.com/paper/2104.07210/) (2021)
- [Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction](https://mlsanity.com/paper/2005.01791/) (2020)
- [Simple Unsupervised Summarization by Contextual Matching](https://mlsanity.com/paper/1907.13337/) (2019)
- [Unsupervised Sentence Compression using Denoising Auto-Encoders](https://mlsanity.com/paper/1809.02669/) (2018)
- [A New Approach to Overgenerating and Scoring Abstractive Summaries](https://mlsanity.com/paper/2104.01726/) (2021)
- [Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks](https://mlsanity.com/paper/2011.02511/) (2020)

## Built on by (most cited first, 12 of 12)

- [SLOG: A Structural Generalization Benchmark for Semantic Parsing](https://mlsanity.com/paper/2310.15040/) (2023)
- [Prompt-Based Editing for Text Style Transfer](https://mlsanity.com/paper/2301.11997/) (2023)
- [Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation](https://mlsanity.com/paper/2210.13800/) (2022)
- [Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2212.10843/) (2022)
- [The Emergence of Chunking Structures with Hierarchical RNN](https://mlsanity.com/paper/2309.04919/) (2023)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation](https://mlsanity.com/paper/2502.04537/) (2025)
- [Evaluating the Efficacy of Length-Controllable Machine Translation](https://mlsanity.com/paper/2305.02300/) (2023)
- [Recent Trends in Unsupervised Summarization](https://mlsanity.com/paper/2305.11231/) (2023)
- [KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generation](https://mlsanity.com/paper/2306.10414/) (2023)
- [From Lengthy to Lucid: A Systematic Literature Review on NLP Techniques for Taming Long Sentences](https://mlsanity.com/paper/2312.05172/) (2023)
- [Query-Guided Self-Supervised Summarization of Nursing Notes](https://mlsanity.com/paper/2407.04125/) (2024)

## Abstract (arXiv)

Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation.