# Efficient Classification of Long Documents Using Transformers

arXiv: https://arxiv.org/abs/2203.11258 · HTML: https://mlsanity.com/paper/2203.11258/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Efficient and Robust Question Answering from Minimal Context over Documents](https://mlsanity.com/paper/1805.08092/) (2018)
- [New Alignment Methods for Discriminative Book Summarization](https://mlsanity.com/paper/1305.1319/) (2013)

## Built on by (most cited first, 30 of 30)

- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Recurrent Attention Networks for Long-text Modeling](https://mlsanity.com/paper/2306.06843/) (2023)
- [Large Language Model Prompt Chaining for Long Legal Document Classification](https://mlsanity.com/paper/2308.04138/) (2023)
- [Breaking the Token Barrier: Chunking and Convolution for Efficient Long Text Classification with BERT](https://mlsanity.com/paper/2310.20558/) (2023)
- [Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks](https://mlsanity.com/paper/2307.07002/) (2023)
- [Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation](https://mlsanity.com/paper/2208.12401/) (2022)
- [RealKIE: Five Novel Datasets for Enterprise Key Information Extraction](https://mlsanity.com/paper/2403.20101/) (2024)
- [Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings](https://mlsanity.com/paper/2305.09316/) (2023)
- [Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study](https://mlsanity.com/paper/2307.09532/) (2023)
- [Attention over pre-trained Sentence Embeddings for Long Document Classification](https://mlsanity.com/paper/2307.09084/) (2023)
- [Hawk: An Industrial-strength Multi-label Document Classifier](https://mlsanity.com/paper/2301.06057/) (2023)
- [Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings?](https://mlsanity.com/paper/2304.14796/) (2023)
- [ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection](https://mlsanity.com/paper/2307.14912/) (2023)
- [LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learning](https://mlsanity.com/paper/2308.01413/) (2023)
- [Leveraging Explainable AI to Analyze Researchers' Aspect-Based Sentiment about ChatGPT](https://mlsanity.com/paper/2308.11001/) (2023)
- [Document Classification using File Names](https://mlsanity.com/paper/2410.01166/) (2024)
- [Improving the Efficiency of Long Document Classification using Sentence Ranking Approach](https://mlsanity.com/paper/2506.07248/) (2025)
- [Quizzard@INOVA Challenge 2025 -- Track A: Plug-and-Play Technique in Interleaved Multi-Image Model](https://mlsanity.com/paper/2506.11737/) (2025)
- [Rethinking Graph-Based Document Classification: Learning Data-Driven Structures Beyond Heuristic Approaches](https://mlsanity.com/paper/2508.00864/) (2025)
- [R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference](https://mlsanity.com/paper/2210.12328/) (2022)
- [Solution for the EPO CodeFest on Green Plastics: Hierarchical multi-label classification of patents relating to green plastics using deep learning](https://mlsanity.com/paper/2302.13784/) (2023)
- [Leveraging BERT Language Model for Arabic Long Document Classification](https://mlsanity.com/paper/2305.03519/) (2023)
- [Predicting Outcomes in Video Games with Long Short Term Memory Networks](https://mlsanity.com/paper/2402.15923/) (2024)
- [Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service](https://mlsanity.com/paper/2403.12563/) (2024)
- [Evaluating Large Language Models for Anxiety and Depression Classification using Counseling and Psychotherapy Transcripts](https://mlsanity.com/paper/2407.13228/) (2024)
- [Domain-specific long text classification from sparse relevant information](https://mlsanity.com/paper/2408.13253/) (2024)
- [Words of War: Exploring the Presidential Rhetorical Arsenal with Deep Learning](https://mlsanity.com/paper/2412.08868/) (2024)
- [Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking](https://mlsanity.com/paper/2508.17490/) (2025)
- [Classifying long legal documents using short random chunks](https://mlsanity.com/paper/2512.24997/) (2025)

## Abstract (arXiv)

Several methods have been proposed for classifying long textual documents using Transformers. However, there is a lack of consensus on a benchmark to enable a fair comparison among different approaches. In this paper, we provide a comprehensive evaluation of the relative efficacy measured against various baselines and diverse datasets -- both in terms of accuracy as well as time and space overheads. Our datasets cover binary, multi-class, and multi-label classification tasks and represent various ways information is organized in a long text (e.g. information that is critical to making the classification decision is at the beginning or towards the end of the document). Our results show that more complex models often fail to outperform simple baselines and yield inconsistent performance across datasets. These findings emphasize the need for future studies to consider comprehensive baselines and datasets that better represent the task of long document classification to develop robust models.