# Hierarchical Learning for Generation with Long Source Sequences

arXiv: https://arxiv.org/abs/2104.07545 · HTML: https://mlsanity.com/paper/2104.07545/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 61 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://mlsanity.com/paper/1809.01576/) (2018)
- [Deep Communicating Agents for Abstractive Summarization](https://mlsanity.com/paper/1803.10357/) (2018)
- [Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization](https://mlsanity.com/paper/1805.05271/) (2018)
- [BP-Transformer: Modelling Long-Range Context via Binary Partitioning](https://mlsanity.com/paper/1911.04070/) (2019)
- [Improving Abstraction in Text Summarization](https://mlsanity.com/paper/1808.07913/) (2018)

## Built on by (most cited first, 30 of 32)

- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022)
- [DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization](https://mlsanity.com/paper/2110.08168/) (2021)
- [An Exploratory Study on Long Dialogue Summarization: What Works and What's Next](https://mlsanity.com/paper/2109.04609/) (2021)
- [Toward Unifying Text Segmentation and Long Document Summarization](https://mlsanity.com/paper/2210.16422/) (2022)
- [HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization](https://mlsanity.com/paper/2203.10741/) (2022)
- [LSG Attention: Extrapolation of pretrained Transformers to long sequences](https://mlsanity.com/paper/2210.15497/) (2022)
- [Leveraging Locality in Abstractive Text Summarization](https://mlsanity.com/paper/2205.12476/) (2022)
- [Abstractive Meeting Summarization: A Survey](https://mlsanity.com/paper/2208.04163/) (2022)
- [GoSum: Extractive Summarization of Long Documents by Reinforcement Learning and Graph Organized discourse state](https://mlsanity.com/paper/2211.10247/) (2022)
- [Enhancing Abstractiveness of Summarization Models through Calibrated Distillation](https://mlsanity.com/paper/2310.13760/) (2023)
- [ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention](https://mlsanity.com/paper/2203.12276/) (2022)
- [A Method to Automate the Discharge Summary Hospital Course for Neurology Patients](https://mlsanity.com/paper/2305.06416/) (2023)
- [TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation](https://mlsanity.com/paper/2206.00847/) (2022)
- [Abstractive Summarization Guided by Latent Hierarchical Document Structure](https://mlsanity.com/paper/2211.09458/) (2022)
- [Learning to Rank Utterances for Query-Focused Meeting Summarization](https://mlsanity.com/paper/2305.12753/) (2023)
- [Meeting Summarization: A Survey of the State of the Art](https://mlsanity.com/paper/2212.08206/) (2022)
- [Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation](https://mlsanity.com/paper/2311.16588/) (2023)
- [EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts](https://mlsanity.com/paper/2204.06604/) (2022)
- [Abstractive Text Summarization Using the BRIO Training Paradigm](https://mlsanity.com/paper/2305.13696/) (2023)
- [Long document summarization using page specific target text alignment and distilling page importance](https://mlsanity.com/paper/2509.16539/) (2025)
- [Document-Level Abstractive Summarization](https://mlsanity.com/paper/2212.03013/) (2022)
- [Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts](https://mlsanity.com/paper/2305.07988/) (2023)
- [SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarization](https://mlsanity.com/paper/2401.17597/) (2024)

## Abstract (arXiv)

One of the challenges for current sequence to sequence (seq2seq) models is processing long sequences, such as those in summarization and document level machine translation tasks. These tasks require the model to reason at the token level as well as the sentence and paragraph level. We design and study a new Hierarchical Attention Transformer-based architecture (HAT) that outperforms standard Transformers on several sequence to sequence tasks. Furthermore, our model achieves state-of-the-art ROUGE scores on four summarization tasks, including PubMed, arXiv, CNN/DM, SAMSum, and AMI. Our model outperforms document-level machine translation baseline on the WMT20 English to German translation task. We investigate what the hierarchical layers learn by visualizing the hierarchical encoder-decoder attention. Finally, we study hierarchical learning on encoder-only pre-training and analyze its performance on classification tasks.