mlsanityData to 14 Oct 2026

A Survey on Long Text Modeling with Transformers

arXiv 2302.14502 ↗First posted 28 Feb 2023cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Modeling long texts has been an essential technique in the field of natural language processing (NLP). With the ever-growing number of long documents, it is important to develop effective modeling methods that can process and analyze such texts. However, long texts pose important research challenges for existing text models, with more complex semantics and special characteristics. In this paper, we provide an overview of the recent advances on long texts modeling based on Transformer models. Firstly, we introduce the formal definition of long text modeling. Then, as the core content, we discuss how to process long input to satisfy the length limitation and design improved Transformer architectures to effectively extend the maximum context length. Following this, we discuss how to adapt Transformer models to capture the special characteristics of long texts. Finally, we describe four typical applications involving long text modeling and conclude this paper with a discussion of future directions. Our survey intends to provide researchers with a synthesis and pointer to related work on long text modeling.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 60 of its 77 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

60 of 77 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
38 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  8. 2021A Survey of Transformerscited by 401
  9. 2021Pay Attention to MLPscited by 368
  10. 2021Random Feature Attentioncited by 354
  11. 2021Efficient Attentions for Long Document Summarizationcited by 345
  12. 2020Sparse Sinkhorn Attentioncited by 251
  13. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  14. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  15. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  16. 2019Hierarchical Transformers for Long Document Classificationcited by 104
  17. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  18. 2021Luna: Linear Unified Nested Attentioncited by 96
  19. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  20. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  21. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  22. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 65
  23. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  24. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  25. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  26. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  27. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  28. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  29. 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
  30. 2021DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarizationcited by 34
  31. 2021Hierarchical Learning for Generation with Long Source Sequencescited by 32
  32. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31
  33. 2022Efficient Classification of Long Documents Using Transformerscited by 30
  34. 2020Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehensioncited by 29
  35. 2022HEGEL: Hypergraph Transformer for Long Document Summarizationcited by 28
  36. 2022Toward Unifying Text Segmentation and Long Document Summarizationcited by 28
  37. 2022HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Informationcited by 26
  38. 2021Long Document Summarization in a Low Resource Setting using Pretrained Language Modelscited by 24

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 26; 2025: 7; 2026: 1.

47 papers.

Sort:

By area:

  1. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  2. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  4. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  5. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  6. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  8. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  9. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  10. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  11. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  12. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
35 more
  1. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  2. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  3. 2024HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systemscited by 37
  4. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
  5. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  6. 2024Grounding Language Model with Chunking-Free In-Context Retrievalcited by 30
  7. 2025RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illuminationcited by 23
  8. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  9. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  10. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  11. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  12. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  13. 2023LogPr'ecis: Unleashing Language Models for Automated Malicious Log Analysiscited by 8
  14. 2023Drilling Down into the Discourse Structure with LLMs for Long Document Question Answeringcited by 8
  15. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7
  16. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  17. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  18. 2024Apollonion: Profile-centric Dialog Agentcited by 4
  19. 2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Modelscited by 4
  20. 2024WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentationcited by 4
  21. 2023LongBoX: Evaluating Transformers on Long-Sequence Clinical Taskscited by 3
  22. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
  23. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  24. 2025Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuningcited by 3
  25. 2023Action-Item-Driven Summarization of Long Meeting Transcriptscited by 2
  26. 2024Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistrycited by 2
  27. 2025CrossFormer: Cross-Segment Semantic Fusion for Document Segmentationcited by 2
  28. 2026Reasoning Fails Where Step Flow Breakscited by 2
  29. 2023Attention Is Not All You Need Anymorecited by 1
  30. 2024Equipping Transformer with Random-Access Reading for Long-Context Understandingcited by 1
  31. 2024Large Language Models and the Extended Church-Turing Thesiscited by 1
  32. 2024Boosting Long-Context Management via Query-Guided Activation Refillingcited by 1
  33. 2024Identifying Narrative Patterns and Outliers in Holocaust Testimonies Using Topic Modelingcited by 0
  34. 2024Large Language Models for Automatic Milestone Detection in Group Discussionscited by 0
  35. 2024Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classificationcited by 0