# Hierarchical Transformers for Multi-Document Summarization

arXiv: https://arxiv.org/abs/1905.13164 · HTML: https://mlsanity.com/paper/1905.13164/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 30 of 132)

- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding](https://mlsanity.com/paper/2105.12723/) (2021)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization](https://mlsanity.com/paper/2110.08499/) (2021)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2401.17043/) (2024)
- [H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences](https://mlsanity.com/paper/2107.11906/) (2021)
- [Exploring Neural Models for Query-Focused Summarization](https://mlsanity.com/paper/2112.07637/) (2021)
- [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022)
- [MReD: A Meta-Review Dataset for Structure-Controllable Text Generation](https://mlsanity.com/paper/2110.07474/) (2021)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://mlsanity.com/paper/2105.14039/) (2021)
- [Models and Datasets for Cross-Lingual Summarisation](https://mlsanity.com/paper/2202.09583/) (2022)
- [TWEETSUMM -- A Dialog Summarization Dataset for Customer Service](https://mlsanity.com/paper/2111.11894/) (2021)
- [Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization](https://mlsanity.com/paper/2208.09770/) (2022)
- [WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus](https://mlsanity.com/paper/2304.04358/) (2023)
- [Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning](https://mlsanity.com/paper/2103.13061/) (2021)
- [Learning to Revise References for Faithful Summarization](https://mlsanity.com/paper/2204.10290/) (2022)
- [Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering](https://mlsanity.com/paper/2305.15387/) (2023)
- [Long-Span Summarization via Local Attention and Content Selection](https://mlsanity.com/paper/2105.03801/) (2021)
- [A Deep Metric Learning Approach to Account Linking](https://mlsanity.com/paper/2105.07263/) (2021)
- [EmailSum: Abstractive Email Thread Summarization](https://mlsanity.com/paper/2107.14691/) (2021)
- [Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search](https://mlsanity.com/paper/2203.08436/) (2022)
- [HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization](https://mlsanity.com/paper/2203.10741/) (2022)

## Abstract (arXiv)

In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial improvements over several strong baselines.