# Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study

arXiv: https://arxiv.org/abs/2307.09532 · HTML: https://mlsanity.com/paper/2307.09532/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Fusing finetuned models for better pretraining](https://mlsanity.com/paper/2204.03044/) (2022)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [Sentiment Analysis Based on Deep Learning: A Comparative Study](https://mlsanity.com/paper/2006.03541/) (2020)
- [MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering](https://mlsanity.com/paper/2010.14095/) (2020)
- [Efficient Classification of Long Documents Using Transformers](https://mlsanity.com/paper/2203.11258/) (2022)
- [Cross-lingual Offensive Language Identification for Low Resource Languages: The Case of Marathi](https://mlsanity.com/paper/2109.03552/) (2021)
- [Comparative Study of Long Document Classification](https://mlsanity.com/paper/2111.00702/) (2021)
- [Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion](https://mlsanity.com/paper/2302.02080/) (2023)
- [Transformers to Fight the COVID-19 Infodemic](https://mlsanity.com/paper/2104.12201/) (2021)
- [A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data](https://mlsanity.com/paper/2106.06738/) (2021)
- [Hierarchical BERT for Medical Document Understanding](https://mlsanity.com/paper/2204.09600/) (2022)
- [WLV-RIT at GermEval 2021: Multitask Learning with Transformers to Detect Toxic, Engaging, and Fact-Claiming Comments](https://mlsanity.com/paper/2108.00057/) (2021)

## Built on by (most cited first, 3 of 3)

- [LLM-based Embedders for Prior Case Retrieval](https://mlsanity.com/paper/2507.18455/) (2025)
- [LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learning](https://mlsanity.com/paper/2308.01413/) (2023)
- [UK-PRBENCH: A Paragraph-Level Precedent Retrieval Benchmark for United Kingdom Case Law](https://mlsanity.com/paper/2609.24613/) (2026)

## Abstract (arXiv)

Text classification is an area of research which has been studied over the years in Natural Language Processing (NLP). Adapting NLP to multiple domains has introduced many new challenges for text classification and one of them is long document classification. While state-of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the transformer models are limited to 512 tokens, and therefore, they struggle with long document classification problems. In this research, we explore on employing Model Fusing for long document classification while comparing the results with well-known BERT and Longformer architectures.