# Self-Supervised Learning for Contextualized Extractive Summarization

arXiv: https://arxiv.org/abs/1906.04466 · HTML: https://mlsanity.com/paper/1906.04466/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Self-Supervised Learning of Graph Neural Networks: A Unified Review](https://mlsanity.com/paper/2102.10757/) (2021)
- [Self-Supervised Representation Learning via Latent Graph Prediction](https://mlsanity.com/paper/2202.08333/) (2022)
- [CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes](https://mlsanity.com/paper/2106.02524/) (2021)
- [Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning](https://mlsanity.com/paper/2204.00166/) (2022)
- [Group Contrastive Self-Supervised Learning on Graphs](https://mlsanity.com/paper/2107.09787/) (2021)
- [Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems via Fine-Grained Contrastive Learning](https://mlsanity.com/paper/2111.10154/) (2021)
- [Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision](https://mlsanity.com/paper/2107.08582/) (2021)
- [Re-entry Prediction for Online Conversations via Self-Supervised Learning](https://mlsanity.com/paper/2109.02020/) (2021)
- [Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00278/) (2022)
- [TegFormer: Topic-to-Essay Generation with Good Topic Coverage and High Text Coherence](https://mlsanity.com/paper/2212.13456/) (2022)
- [Inducing Causal Structure for Abstractive Text Summarization](https://mlsanity.com/paper/2308.12888/) (2023)

## Abstract (arXiv)

Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level. In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion. Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks. Furthermore, we show that after pre-training, a clean model with simple building blocks is able to outperform previous state-of-the-art that are carefully designed.