# How to Fine-Tune BERT for Text Classification?

arXiv: https://arxiv.org/abs/1905.05583 · HTML: https://mlsanity.com/paper/1905.05583/

## Key facts

- Cited by 411 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 30 of 411)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome](https://mlsanity.com/paper/2306.15006/) (2023)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning](https://mlsanity.com/paper/2302.13485/) (2023)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [The Limits of Word Level Differential Privacy](https://mlsanity.com/paper/2205.02130/) (2022)
- [CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems](https://mlsanity.com/paper/2103.15721/) (2021)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better](https://mlsanity.com/paper/2202.12024/) (2022)
- [Self-Tuning for Data-Efficient Deep Learning](https://mlsanity.com/paper/2102.12903/) (2021)
- [Improved Text Classification via Contrastive Adversarial Training](https://mlsanity.com/paper/2107.10137/) (2021)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [Scaling Federated Learning for Fine-tuning of Large Language Models](https://mlsanity.com/paper/2102.00875/) (2021)
- [KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2402.15043/) (2024)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)

## Abstract (arXiv)

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets.