# GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization

arXiv: https://arxiv.org/abs/2208.09982 · HTML: https://mlsanity.com/paper/2208.09982/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Focus Attention: Promoting Faithfulness and Diversity in Summarization](https://mlsanity.com/paper/2105.11921/) (2021)
- [Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks](https://mlsanity.com/paper/2010.06253/) (2020)
- [Enriching and Controlling Global Semantics for Text Summarization](https://mlsanity.com/paper/2109.10616/) (2021)
- [Topic-Guided Abstractive Multi-Document Summarization](https://mlsanity.com/paper/2110.11207/) (2021)
- [Stepwise Extractive Summarization and Planning with Structured Transformers](https://mlsanity.com/paper/2010.02744/) (2020)
- [HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization](https://mlsanity.com/paper/2110.06388/) (2021)
- [Topic representation: finding more representative words in topic models](https://mlsanity.com/paper/1810.10307/) (2018)

## Built on by (most cited first, 10 of 10)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Readability Controllable Biomedical Document Summarization](https://mlsanity.com/paper/2210.04705/) (2022)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling](https://mlsanity.com/paper/2305.11719/) (2023)
- [Contrastive Hierarchical Discourse Graph for Scientific Document Summarization](https://mlsanity.com/paper/2306.00177/) (2023)
- [Diversity Enhances an LLM's Performance in RAG and Long-context Task](https://mlsanity.com/paper/2502.09017/) (2025)
- [Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://mlsanity.com/paper/2306.01169/) (2023)
- [A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models](https://mlsanity.com/paper/2304.08763/) (2023)
- [State Space Models for Extractive Summarization in Low Resource Scenarios](https://mlsanity.com/paper/2501.14673/) (2025)

## Abstract (arXiv)

Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global semantic information for text summarization. However, in these methods, there remain limitations in the way they capture and integrate the global semantic information. In this paper, we propose a novel model, the graph contrastive topic enhanced language model (GRETEL), that incorporates the graph contrastive topic model with the pre-trained language model, to fully leverage both the global and local contextual semantics for long document extractive summarization. To better capture and incorporate the global semantic information into PLMs, the graph contrastive topic model integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. To this end, GRETEL encourages the model to efficiently extract salient sentences that are topically related to the gold summary, rather than redundant sentences that cover sub-optimal topics. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.