# DocBERT: BERT for Document Classification

arXiv: https://arxiv.org/abs/1904.08398 · HTML: https://mlsanity.com/paper/1904.08398/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 30 of 68)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patients](https://mlsanity.com/paper/2309.12625/) (2023)
- [Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet](https://mlsanity.com/paper/2102.08036/) (2021)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [FakeFlow: Fake News Detection by Modeling the Flow of Affective Information](https://mlsanity.com/paper/2101.09810/) (2021)
- [Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosis](https://mlsanity.com/paper/2108.07392/) (2021)
- [Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning](https://mlsanity.com/paper/2106.03103/) (2021)
- ["Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groups](https://mlsanity.com/paper/2102.03870/) (2021)
- [Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations](https://mlsanity.com/paper/2102.11573/) (2021)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [Comparative Study of Long Document Classification](https://mlsanity.com/paper/2111.00702/) (2021)
- [You too Brutus! Trapping Hateful Users in Social Media: Challenges, Solutions & Insights](https://mlsanity.com/paper/2108.00524/) (2021)
- [Improving Short Text Classification With Augmented Data Using GPT-3](https://mlsanity.com/paper/2205.10981/) (2022)
- [Comparison of Czech Transformers on Text Classification Tasks](https://mlsanity.com/paper/2107.10042/) (2021)
- [FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning](https://mlsanity.com/paper/2310.16538/) (2023)
- [Automated Identification of Toxic Code Reviews Using ToxiCR](https://mlsanity.com/paper/2202.13056/) (2022)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [Unsupervised Matching of Data and Text](https://mlsanity.com/paper/2112.08776/) (2021)
- [Hierarchical Neural Network Approaches for Long Document Classification](https://mlsanity.com/paper/2201.06774/) (2022)
- [Less is More: Parameter-Free Text Classification with Gzip](https://mlsanity.com/paper/2212.09410/) (2022)
- [Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification](https://mlsanity.com/paper/2104.08444/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation](https://mlsanity.com/paper/2112.05364/) (2021)
- [Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models](https://mlsanity.com/paper/2204.04289/) (2022)
- [Protoformer: Embedding Prototypes for Transformers](https://mlsanity.com/paper/2206.12710/) (2022)
- [ATLAS: Automatically Detecting Discrepancies Between Privacy Policies and Privacy Labels](https://mlsanity.com/paper/2306.09247/) (2023)

## Abstract (arXiv)

We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.