# Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition

arXiv: https://arxiv.org/abs/2202.02617 · HTML: https://mlsanity.com/paper/2202.02617/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 5 Feb 2022.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models](https://mlsanity.com/paper/1909.11299/) (2019)
- [Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://mlsanity.com/paper/2007.01658/) (2020)

## Built on by (most cited first, 3 of 3)

- [The Role of Feedback Alignment in Self-Distillation](https://mlsanity.com/paper/2606.11173/) (2026)
- [A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initialization](https://mlsanity.com/paper/2602.13942/) (2026)
- [Read, Extract, Classify: A Tool for Smarter Requirements Engineering](https://mlsanity.com/paper/2605.11045/) (2026)

## Abstract (arXiv)

The current standard approach for fine-tuning transformer-based language models includes a fixed number of training epochs and a linear learning rate schedule. In order to obtain a near-optimal model for the given downstream task, a search in optimization hyperparameter space is usually required. In particular, the number of training epochs needs to be adjusted to the dataset size. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule to dynamically adjust the number of training epochs to the dataset size. For the example use case of named entity recognition, we show that our approach not only makes hyperparameter search with respect to the number of training epochs redundant, but also leads to improved results in terms of performance, stability and efficiency. This holds true especially for small datasets, where we outperform the state-of-the-art fine-tuning method by a large margin.