mlsanityData to 7 Oct 2026

Hierarchical Learning for Generation with Long Source Sequences

arXiv 2104.07545 ↗First posted 15 Apr 2021cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

One of the challenges for current sequence to sequence (seq2seq) models is processing long sequences, such as those in summarization and document level machine translation tasks. These tasks require the model to reason at the token level as well as the sentence and paragraph level. We design and study a new Hierarchical Attention Transformer-based architecture (HAT) that outperforms standard Transformers on several sequence to sequence tasks. Furthermore, our model achieves state-of-the-art ROUGE scores on four summarization tasks, including PubMed, arXiv, CNN/DM, SAMSum, and AMI. Our model outperforms document-level machine translation baseline on the WMT20 English to German translation task. We investigate what the hierarchical layers learn by visualizing the hierarchical encoder-decoder attention. Finally, we study hierarchical learning on encoder-only pre-training and analyze its performance on classification tasks.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

36 of 61 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
24 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2015Teaching Machines to Read and Comprehendcited by 1,039
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  12. 2021Efficient Attentions for Long Document Summarizationcited by 342
  13. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  14. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  15. 2018Deep Communicating Agents for Abstractive Summarizationcited by 50
  16. 2018Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximizationcited by 46
  17. 2019BP-Transformer: Modelling Long-Range Context via Binary Partitioningcited by 38
  18. 2018Improving Abstraction in Text Summarizationcited by 35
  19. 2017Hierarchical Recurrent Attention Network for Response Generationcited by 34
  20. 2019Multi-Agent Actor-Critic with Hierarchical Graph Attention Networkcited by 33
  21. 2020Disentangled Variational Autoencoder based Multi-Label Classification with Covariance-Aware Multivariate Probit Modelcited by 22
  22. 2020Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Gamescited by 22
  23. 2020Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarizationcited by 13
  24. 2017Deconvolutional Paragraph Representation Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 14; 2023: 8; 2024: 2; 2025: 2.

32 papers.

Sort:

By area:

  1. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  2. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  3. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  4. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  5. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  6. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  7. 2023A Survey on Long Text Modeling with Transformerscited by 47
  8. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  9. 2021DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarizationcited by 34
  10. 2021An Exploratory Study on Long Dialogue Summarization: What Works and What's Nextcited by 28
  11. 2022Toward Unifying Text Segmentation and Long Document Summarizationcited by 28
  12. 2022HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarizationcited by 22
20 more
  1. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  2. 2022Leveraging Locality in Abstractive Text Summarizationcited by 9
  3. 2022Abstractive Meeting Summarization: A Surveycited by 9
  4. 2022GoSum: Extractive Summarization of Long Documents by Reinforcement Learning and Graph Organized discourse statecited by 9
  5. 2023Enhancing Abstractiveness of Summarization Models through Calibrated Distillationcited by 8
  6. 2022ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attentioncited by 6
  7. 2023A Method to Automate the Discharge Summary Hospital Course for Neurology Patientscited by 6
  8. 2022TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generationcited by 4
  9. 2022Abstractive Summarization Guided by Latent Hierarchical Document Structurecited by 4
  10. 2023Learning to Rank Utterances for Query-Focused Meeting Summarizationcited by 4
  11. 2022Meeting Summarization: A Survey of the State of the Artcited by 3
  12. 2023Ascle: A Python Natural Language Processing Toolkit for Medical Text Generationcited by 3
  13. 2022EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Textscited by 1
  14. 2023Abstractive Text Summarization Using the BRIO Training Paradigmcited by 1
  15. 2025Long document summarization using page specific target text alignment and distilling page importancecited by 1
  16. 2022Document-Level Abstractive Summarizationcited by 0
  17. 2023Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcriptscited by 0
  18. 2024SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarizationcited by 0
  19. 2024Investigating Consistency in Query-Based Meeting Summarization: A Comparative Study of Different Embedding Methodscited by 0
  20. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0