# Comparative Study of Long Document Classification

arXiv: https://arxiv.org/abs/2111.00702 · HTML: https://mlsanity.com/paper/2111.00702/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [DocBERT: BERT for Document Classification](https://mlsanity.com/paper/1904.08398/) (2019)
- [Comparing BERT against traditional machine learning text classification](https://mlsanity.com/paper/2005.13012/) (2020)
- [Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks](https://mlsanity.com/paper/1603.03827/) (2016)
- [Deep Learning for Hindi Text Classification: A Comparison](https://mlsanity.com/paper/2001.10340/) (2020)
- [Experimental Evaluation of Deep Learning models for Marathi Text Classification](https://mlsanity.com/paper/2101.04899/) (2021)
- [Evaluating Deep Learning Approaches for Covid19 Fake News Detection](https://mlsanity.com/paper/2101.04012/) (2021)
- [ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models](https://mlsanity.com/paper/2102.00238/) (2021)

## Built on by (most cited first, 12 of 12)

- [Hierarchical Neural Network Approaches for Long Document Classification](https://mlsanity.com/paper/2201.06774/) (2022)
- [Large Language Model Prompt Chaining for Long Legal Document Classification](https://mlsanity.com/paper/2308.04138/) (2023)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)
- [Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study](https://mlsanity.com/paper/2307.09532/) (2023)
- [On Sensitivity of Deep Learning Based Text Classification Algorithms to Practical Input Perturbations](https://mlsanity.com/paper/2201.00318/) (2022)
- [Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification](https://mlsanity.com/paper/2209.12943/) (2022)
- [Hawk: An Industrial-strength Multi-label Document Classifier](https://mlsanity.com/paper/2301.06057/) (2023)
- [Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text](https://mlsanity.com/paper/2303.17786/) (2023)
- [Improving the Efficiency of Long Document Classification using Sentence Ranking Approach](https://mlsanity.com/paper/2506.07248/) (2025)
- [Leveraging BERT Language Model for Arabic Long Document Classification](https://mlsanity.com/paper/2305.03519/) (2023)
- [Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking](https://mlsanity.com/paper/2508.17490/) (2025)
- [Classifying long legal documents using short random chunks](https://mlsanity.com/paper/2512.24997/) (2025)

## Abstract (arXiv)

The amount of information stored in the form of documents on the internet has been increasing rapidly. Thus it has become a necessity to organize and maintain these documents in an optimum manner. Text classification algorithms study the complex relationships between words in a text and try to interpret the semantics of the document. These algorithms have evolved significantly in the past few years. There has been a lot of progress from simple machine learning algorithms to transformer-based architectures. However, existing literature has analyzed different approaches on different data sets thus making it difficult to compare the performance of machine learning algorithms. In this work, we revisit long document classification using standard machine learning approaches. We benchmark approaches ranging from simple Naive Bayes to complex BERT on six standard text classification datasets. We present an exhaustive comparison of different algorithms on a range of long document datasets. We re-iterate that long document classification is a simpler task and even basic algorithms perform competitively with BERT-based approaches on most of the datasets. The BERT-based models perform consistently well on all the datasets and can be blindly used for the document classification task when the computations cost is not a concern. In the shallow model's category, we suggest the usage of raw BiLSTM + Max architecture which performs decently across all the datasets. Even simpler Glove + Attention bag of words model can be utilized for simpler use cases. The importance of using sophisticated models is clearly visible in the IMDB sentiment dataset which is a comparatively harder task.