mlsanityData to 7 Oct 2026

Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization

arXiv 2305.16259 ↗First posted 25 May 2023cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural Language Processing (NLP) during the past decade. However, the demands of long document analysis are quite different from those of shorter texts, while the ever increasing size of documents uploaded online renders automated understanding of lengthy texts a critical issue. Relevant applications include automated Web mining, legal document review, medical records analysis, financial reports analysis, contract management, environmental impact assessment, news aggregation, etc. Despite the relatively recent development of efficient algorithms for analyzing long documents, practical tools in this field are currently flourishing. This article serves as an entry point into this dynamic domain and aims to achieve two objectives. First of all, it provides an introductory overview of the relevant neural building blocks, serving as a concise tutorial for the field. Secondly, it offers a brief examination of the current state-of-the-art in two key long document analysis tasks: document classification and document summarization. Sentiment analysis for long texts is also covered, since it is typically treated as a particular case of document classification. Consequently, this article presents an introductory exploration of document-level analysis, addressing the primary challenges, concerns, and existing solutions. Finally, it offers a concise definition of "long text/document", presents an original overarching taxonomy of common deep neural methods for long document analysis and lists publicly available annotated datasets that can facilitate further research in this area.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 50 of its 112 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

50 of 112 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2014Distributed Representations of Sentences and Documentscited by 677
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  10. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  11. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  12. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 245
38 more
  1. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  2. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  3. 2020Extractive Summarization as Text Matchingcited by 189
  4. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  5. 2019Text Classification Algorithms: A Surveycited by 163
  6. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  7. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  8. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  9. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  10. 2016Neural Summarization by Extracting Sentences and Wordscited by 114
  11. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  12. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  13. 2020A Survey on Text Classification: From Shallow to Deep Learningcited by 80
  14. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  15. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  16. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  17. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  18. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  19. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  20. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  21. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  22. 2019An Annotated Dataset of Coreference in English Literaturecited by 57
  23. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  24. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  25. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  26. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  27. 2020An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labelscited by 44
  28. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  29. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  30. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31
  31. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  32. 2022HEGEL: Hypergraph Transformer for Long Document Summarizationcited by 28
  33. 2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24
  34. 2022Improving the Faithfulness of Abstractive Summarization via Entity Coverage Controlcited by 21
  35. 2022Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistencycited by 11
  36. 2022Hierarchical Neural Network Approaches for Long Document Classificationcited by 7
  37. 2021Sentiment analysis in tweets: an assessment study from classical to modern text representation modelscited by 3
  38. 2021The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrievalcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 2.

7 papers.

  1. 2024Synthetic4Health: Generating Annotated Synthetic Clinical Letterscited by 3
  2. 2025Designing Synthetic Discussion Generation Systems: A Case Study for Online Facilitationcited by 2
  3. 2024Leveraging Digital Twin Technologies for Public Space Protection and Vulnerability Assessmentcited by 1
  4. 2026Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinionscited by 1
  5. 2024Entity Extraction from High-Level Corruption Schemes via Large Language Modelscited by 0
  6. 2024A Study on Context Length and Efficient Transformers for Biomedical Image Analysiscited by 0
  7. 2026Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurementcited by 0