# Stepwise Extractive Summarization and Planning with Structured Transformers

arXiv: https://arxiv.org/abs/2010.02744 · HTML: https://mlsanity.com/paper/2010.02744/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [Toward Unifying Text Segmentation and Long Document Summarization](https://mlsanity.com/paper/2210.16422/) (2022)
- [Long-Span Summarization via Local Attention and Content Selection](https://mlsanity.com/paper/2105.03801/) (2021)
- [Enriching and Controlling Global Semantics for Text Summarization](https://mlsanity.com/paper/2109.10616/) (2021)
- [Data-to-text Generation with Variational Sequential Planning](https://mlsanity.com/paper/2202.13756/) (2022)
- [HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization](https://mlsanity.com/paper/2110.06388/) (2021)
- [TLDR9+: A Large Scale Resource for Extreme Summarization of Social Media Posts](https://mlsanity.com/paper/2110.01159/) (2021)
- [GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization](https://mlsanity.com/paper/2208.09982/) (2022)
- [OTExtSum: Extractive Text Summarisation with Optimal Transport](https://mlsanity.com/paper/2204.10086/) (2022)
- [CoPERLex: Content Planning with Event-based Representations for Legal Case Summarization](https://mlsanity.com/paper/2501.14112/) (2025)
- [Multilingual Natural Language Processing Model for Radiology Reports -- The Summary is all you need!](https://mlsanity.com/paper/2310.00100/) (2023)
- [Curriculum-Guided Abstractive Summarization](https://mlsanity.com/paper/2302.01342/) (2023)
- [State Space Models for Extractive Summarization in Low Resource Scenarios](https://mlsanity.com/paper/2501.14673/) (2025)
- [Data Transformation Strategies to Remove Heterogeneity](https://mlsanity.com/paper/2507.12677/) (2025)

## Abstract (arXiv)

We propose encoder-centric stepwise models for extractive summarization using structured transformers -- HiBERT and Extended Transformers. We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure. Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive content planner for different tasks. When evaluated on CNN/DailyMail extractive summarization, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filtering. This also holds true for Rotowire table-to-text generation, where our models surpass previously reported metrics for content selection, planning and ordering, highlighting the strength of stepwise modeling. Amongst the two structured transformers we test, stepwise Extended Transformers provides the best performance across both datasets and sets a new standard for these challenges.