# To Tune or Not To Tune? How About the Best of Both Worlds?

arXiv: https://arxiv.org/abs/1907.05338 · HTML: https://mlsanity.com/paper/1907.05338/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2019.

## Built on by (most cited first, 4 of 4)

- [Self-Tuning for Data-Efficient Deep Learning](https://mlsanity.com/paper/2102.12903/) (2021)
- [Cluster & Tune: Boost Cold Start Performance in Text Classification](https://mlsanity.com/paper/2203.10581/) (2022)
- [Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain](https://mlsanity.com/paper/2103.02691/) (2021)
- [Continuous Multi-Task Pre-training for Malicious URL Detection and Webpage Classification](https://mlsanity.com/paper/2402.11495/) (2024)

## Abstract (arXiv)

The introduction of pre-trained language models has revolutionized natural language research communities. However, researchers still know relatively little regarding their theoretical and empirical properties. In this regard, Peters et al. perform several experiments which demonstrate that it is better to adapt BERT with a light-weight task-specific head, rather than building a complex one on top of the pre-trained language model, and freeze the parameters in the said language model. However, there is another option to adopt. In this paper, we propose a new adaptation method which we first train the task model with the BERT parameters frozen and then fine-tune the entire model together. Our experimental results show that our model adaptation method can achieve 4.7% accuracy improvement in semantic similarity task, 0.99% accuracy improvement in sequence labeling task and 0.72% accuracy improvement in the text classification task.