# HEGEL: Hypergraph Transformer for Long Document Summarization

arXiv: https://arxiv.org/abs/2210.04126 · HTML: https://mlsanity.com/paper/2210.04126/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Graph-Bert: Only Attention is Needed for Learning Graph Representations](https://mlsanity.com/paper/2001.05140/) (2020)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://mlsanity.com/paper/1802.08636/) (2018)
- [Neural Summarization by Extracting Sentences and Words](https://mlsanity.com/paper/1603.07252/) (2016)
- [Heterogeneous Graph Neural Networks for Extractive Document Summarization](https://mlsanity.com/paper/2004.12393/) (2020)
- [On Extractive and Abstractive Neural Document Summarization with Transformer Language Models](https://mlsanity.com/paper/1909.03186/) (2019)
- [Be More with Less: Hypergraph Attention Networks for Inductive Text Classification](https://mlsanity.com/paper/2011.00387/) (2020)
- [Sentence Centrality Revisited for Unsupervised Summarization](https://mlsanity.com/paper/1906.03508/) (2019)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [Graph-based Neural Multi-Document Summarization](https://mlsanity.com/paper/1706.06681/) (2017)
- [HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information](https://mlsanity.com/paper/2203.09629/) (2022)
- [Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks](https://mlsanity.com/paper/2010.06253/) (2020)
- [Systematically Exploring Redundancy Reduction in Summarizing Long Documents](https://mlsanity.com/paper/2012.00052/) (2020)
- [Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control](https://mlsanity.com/paper/2207.02263/) (2022)
- [Multiplex Graph Neural Network for Extractive Text Summarization](https://mlsanity.com/paper/2108.12870/) (2021)
- [Topic-Guided Abstractive Multi-Document Summarization](https://mlsanity.com/paper/2110.11207/) (2021)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)

## Built on by (most cited first, 28 of 28)

- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023)
- [SummIt: Iterative Text Summarization via ChatGPT](https://mlsanity.com/paper/2305.14835/) (2023)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [DiffuSum: Generation Enhanced Extractive Summarization with Diffusion](https://mlsanity.com/paper/2305.01735/) (2023)
- [Contrastive Hierarchical Discourse Graph for Scientific Document Summarization](https://mlsanity.com/paper/2306.00177/) (2023)
- [Computing and Exploiting Document Structure to Improve Unsupervised Extractive Summarization of Legal Case Decisions](https://mlsanity.com/paper/2211.03229/) (2022)
- [Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization](https://mlsanity.com/paper/2305.16259/) (2023)
- [StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMs](https://mlsanity.com/paper/2505.22950/) (2025)
- [HDT: Hierarchical Document Transformer](https://mlsanity.com/paper/2407.08330/) (2024)
- [Inverse Reinforcement Learning for Text Summarization](https://mlsanity.com/paper/2212.09917/) (2022)
- [Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://mlsanity.com/paper/2306.01169/) (2023)
- [Core Building Blocks: Next Gen Geo Spatial GPT Application](https://mlsanity.com/paper/2310.11029/) (2023)
- [Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Level](https://mlsanity.com/paper/2405.10202/) (2024)
- [PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference](https://mlsanity.com/paper/2409.15104/) (2024)
- [DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization](https://mlsanity.com/paper/2410.15687/) (2024)
- [GraphLSS: Integrating Lexical, Structural, and Semantic Features for Long Document Extractive Summarization](https://mlsanity.com/paper/2410.21315/) (2024)
- [Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning](https://mlsanity.com/paper/2605.27259/) (2026)
- [Clinical Note Owns its Hierarchy: Multi-Level Hypergraph Neural Networks for Patient-Level Representation Learning](https://mlsanity.com/paper/2305.09756/) (2023)
- [Unsupervised Multi-document Summarization with Holistic Inference](https://mlsanity.com/paper/2309.04087/) (2023)
- [Topology-guided Hypergraph Transformer Network: Unveiling Structural Insights for Improved Representation](https://mlsanity.com/paper/2310.09657/) (2023)
- [Learning Co-Speech Gesture for Multimodal Aphasia Type Detection](https://mlsanity.com/paper/2310.11710/) (2023)
- [NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents](https://mlsanity.com/paper/2402.17682/) (2024)
- [ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference](https://mlsanity.com/paper/2509.03565/) (2025)
- [Prototype-Enhanced Hypergraph Learning for Heterogeneous Information Networks](https://mlsanity.com/paper/2309.13092/) (2023)
- [T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task](https://mlsanity.com/paper/2409.17640/) (2024)
- [StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training](https://mlsanity.com/paper/2411.16618/) (2024)

## Abstract (arXiv)

Extractive summarization for long documents is challenging due to the extended structured input context. The long-distance sentence dependency hinders cross-sentence relations modeling, the critical step of extractive summarization. This paper proposes HEGEL, a hypergraph neural network for long document summarization by capturing high-order cross-sentence relations. HEGEL updates and learns effective sentence representations with hypergraph transformer layers and fuses different types of sentence dependencies, including latent topics, keywords coreference, and section structure. We validate HEGEL by conducting extensive experiments on two benchmark datasets, and experimental results demonstrate the effectiveness and efficiency of HEGEL.