# Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization arXiv: https://arxiv.org/abs/2305.16259 ยท HTML: https://mlsanity.com/paper/2305.16259/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 50 of its 112 references that are in the map. - First posted on arXiv 25 May 2023. ## Builds on (most cited first, 30 of 50) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019) - [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020) - [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014) - [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020) - [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019) - [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021) - [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016) - [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021) - [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021) - [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020) - [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021) - [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019) - [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021) - [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021) - [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021) - [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021) - [Neural Summarization by Extracting Sentences and Words](https://mlsanity.com/paper/1603.07252/) (2016) - [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020) - [Fill in the BLANC: Human-free quality estimation of document summaries](https://mlsanity.com/paper/2002.09836/) (2020) - [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020) - [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021) - [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022) - [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://mlsanity.com/paper/2004.03066/) (2020) - [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022) - [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021) ## Built on by (most cited first, 7 of 7) - [Synthetic4Health: Generating Annotated Synthetic Clinical Letters](https://mlsanity.com/paper/2409.09501/) (2024) - [Designing Synthetic Discussion Generation Systems: A Case Study for Online Facilitation](https://mlsanity.com/paper/2503.16505/) (2025) - [Leveraging Digital Twin Technologies for Public Space Protection and Vulnerability Assessment](https://mlsanity.com/paper/2408.17136/) (2024) - [Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions](https://mlsanity.com/paper/2604.12138/) (2026) - [Entity Extraction from High-Level Corruption Schemes via Large Language Models](https://mlsanity.com/paper/2409.13704/) (2024) - [A Study on Context Length and Efficient Transformers for Biomedical Image Analysis](https://mlsanity.com/paper/2501.00619/) (2024) - [Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement](https://mlsanity.com/paper/2609.27165/) (2026) ## Abstract (arXiv) The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural Language Processing (NLP) during the past decade. However, the demands of long document analysis are quite different from those of shorter texts, while the ever increasing size of documents uploaded online renders automated understanding of lengthy texts a critical issue. Relevant applications include automated Web mining, legal document review, medical records analysis, financial reports analysis, contract management, environmental impact assessment, news aggregation, etc. Despite the relatively recent development of efficient algorithms for analyzing long documents, practical tools in this field are currently flourishing. This article serves as an entry point into this dynamic domain and aims to achieve two objectives. First of all, it provides an introductory overview of the relevant neural building blocks, serving as a concise tutorial for the field. Secondly, it offers a brief examination of the current state-of-the-art in two key long document analysis tasks: document classification and document summarization. Sentiment analysis for long texts is also covered, since it is typically treated as a particular case of document classification. Consequently, this article presents an introductory exploration of document-level analysis, addressing the primary challenges, concerns, and existing solutions. Finally, it offers a concise definition of "long text/document", presents an original overarching taxonomy of common deep neural methods for long document analysis and lists publicly available annotated datasets that can facilitate further research in this area.