mlsanityData to 13 Oct 2026

LongT5: Efficient Text-To-Text Transformer for Long Sequences

arXiv 2112.07916 ↗First posted 15 Dec 2021cs.CLcited by 232

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call em Transient Global (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

Key facts. Cited by 232 papers in the map (data through 13 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 15 Dec 2021.

Builds on

35 of 43 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2020Scaling Laws for Neural Language Modelscited by 6,494
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  7. 2020Longformer: The Long-Document Transformercited by 3,393
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  10. 2020Rethinking Attention with Performerscited by 1,594
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,206
23 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,093
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  5. 2020Efficient Transformers: A Surveycited by 785
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 697
  7. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 526
  8. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  9. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 429
  10. 2021Random Feature Attentioncited by 351
  11. 2020Sparse Sinkhorn Attentioncited by 251
  12. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  13. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  14. 2021Luna: Linear Unified Nested Attentioncited by 96
  15. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  16. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  17. 2021Hierarchical Learning for Generation with Long Source Sequencescited by 32
  18. 2018Word Mover's Embedding: From Word2Vec to Document Embeddingcited by 26
  19. 2021Topic-Guided Abstractive Multi-Document Summarizationcited by 19
  20. 2017Efficient Vector Representation for Documents through Corruptioncited by 16
  21. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  22. 2019Hierarchical Document Encoder for Parallel Corpus Miningcited by 10
  23. 2016Representing Documents and Queries as Sets of Word Embedded Vectors for Information Retrievalcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 39; 2023: 69; 2024: 69; 2025: 43; 2026: 12.

232 papers.

Sort:

By area:

  1. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,354
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 820
  3. 2023A Comprehensive Overview of Large Language Modelscited by 644
  4. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 527
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 515
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 351
  8. 2022UL2: Unifying Language Learning Paradigmscited by 294
  9. 2023Language Modeling Is Compressioncited by 225
  10. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 190
  11. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  12. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 156
38 more
  1. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 144
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  3. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 132
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 104
  7. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  9. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Frameworkcited by 98
  10. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  11. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  12. 2022Parallel Context Windows for Large Language Modelscited by 81
  13. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  14. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  15. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  16. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  17. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  18. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  19. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 63
  20. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  21. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  22. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 54
  23. 2023A Survey on Long Text Modeling with Transformerscited by 47
  24. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  25. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  26. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  27. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  28. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  29. 2022Prompt Injection: Parameterization of Fixed Inputscited by 35
  30. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  31. 2022MOPRD: A multidisciplinary open peer review datasetcited by 32
  32. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  33. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  34. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  35. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  36. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  37. 2023QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operationscited by 30
  38. 2022Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarizationcited by 29

50 most cited of 232 All 232 (JSON)