# Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking

arXiv: https://arxiv.org/abs/2508.17490 · HTML: https://mlsanity.com/paper/2508.17490/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 24 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [Sparse and Continuous Attention Mechanisms](https://mlsanity.com/paper/2006.07214/) (2020)
- [Efficient Classification of Long Documents Using Transformers](https://mlsanity.com/paper/2203.11258/) (2022)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [New Alignment Methods for Discriminative Book Summarization](https://mlsanity.com/paper/1305.1319/) (2013)
- [An Improved Text Sentiment Classification Model Using TF-IDF and Next Word Negation](https://mlsanity.com/paper/1806.06407/) (2018)
- [Comparative Study of Long Document Classification](https://mlsanity.com/paper/2111.00702/) (2021)
- [Multi-class Text Classification using BERT-based Active Learning](https://mlsanity.com/paper/2104.14289/) (2021)
- [A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset](https://mlsanity.com/paper/2308.04037/) (2023)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [Hierarchical Neural Network Approaches for Long Document Classification](https://mlsanity.com/paper/2201.06774/) (2022)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)
- [Improving the Efficiency of Long Document Classification using Sentence Ranking Approach](https://mlsanity.com/paper/2506.07248/) (2025)

## Abstract (arXiv)

Transformer-based models like BERT excel at short text classification but struggle with long document classification (LDC) due to input length limitations and computational inefficiencies. In this work, we propose an efficient, zero-shot approach to LDC that leverages sentence ranking to reduce input context without altering the model architecture. Our method enables the adaptation of models trained on short texts, such as headlines, to long-form documents by selecting the most informative sentences using a TF-IDF-based ranking strategy. Using the MahaNews dataset of long Marathi news articles, we evaluate three context reduction strategies that prioritize essential content while preserving classification accuracy. Our results show that retaining only the top 50% ranked sentences maintains performance comparable to full-document inference while reducing inference time by up to 35%. This demonstrates that sentence ranking is a simple yet effective technique for scalable and efficient zero-shot LDC.