mlsanityData to 14 Oct 2026

Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study

arXiv 2307.09532 ↗First posted 18 Jul 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Text classification is an area of research which has been studied over the years in Natural Language Processing (NLP). Adapting NLP to multiple domains has introduced many new challenges for text classification and one of them is long document classification. While state-of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the transformer models are limited to 512 tokens, and therefore, they struggle with long document classification problems. In this research, we explore on employing Model Fusing for long document classification while comparing the results with well-known BERT and Longformer architectures.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

19 of 36 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  4. 2019Neural Legal Judgment Prediction in Englishcited by 209
  5. 2019Text Classification Algorithms: A Surveycited by 163
  6. 2022Fusing finetuned models for better pretrainingcited by 110
  7. 2019Hierarchical Transformers for Long Document Classificationcited by 104
  8. 2020Multilingual Offensive Language Identification with Cross-lingual Embeddingscited by 71
  9. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  10. 2020Sentiment Analysis Based on Deep Learning: A Comparative Studycited by 37
  11. 2020MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answeringcited by 30
  12. 2022Efficient Classification of Long Documents Using Transformerscited by 30
7 more
  1. 2021Cross-lingual Offensive Language Identification for Low Resource Languages: The Case of Marathicited by 28
  2. 2021Comparative Study of Long Document Classificationcited by 12
  3. 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
  4. 2021Transformers to Fight the COVID-19 Infodemiccited by 4
  5. 2021A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Datacited by 4
  6. 2022Hierarchical BERT for Medical Document Understandingcited by 4
  7. 2021WLV-RIT at GermEval 2021: Multitask Learning with Transformers to Detect Toxic, Engaging, and Fact-Claiming Commentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2025LLM-based Embedders for Prior Case Retrievalcited by 2
  2. 2023LaFiCMIL: Rethinking Large File Classification from the Perspective of Correlated Multiple Instance Learningcited by 1
  3. 2026UK-PRBENCH: A Paragraph-Level Precedent Retrieval Benchmark for United Kingdom Case Lawcited by 0