# Gradual Fine-Tuning for Low-Resource Domain Adaptation

arXiv: https://arxiv.org/abs/2103.02205 · HTML: https://mlsanity.com/paper/2103.02205/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://mlsanity.com/paper/1907.07421/) (2019)
- [Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/1905.05816/) (2019)
- [Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus](https://mlsanity.com/paper/2004.06295/) (2020)
- [Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back Translation](https://mlsanity.com/paper/1910.03009/) (2019)
- [Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1907.03060/) (2019)
- [Cross-lingual Semantic Role Labeling with Model Transfer](https://mlsanity.com/paper/2008.10284/) (2020)
- [Question Answering through Transfer Learning from Large Fine-grained Supervision Data](https://mlsanity.com/paper/1702.02171/) (2017)
- [Translate and Label! An Encoder-Decoder Approach for Cross-lingual Semantic Role Labeling](https://mlsanity.com/paper/1908.11326/) (2019)

## Built on by (most cited first, 24 of 24)

- [NICO++: Towards Better Benchmarking for Domain Generalization](https://mlsanity.com/paper/2204.08040/) (2022)
- [Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction](https://mlsanity.com/paper/2203.08308/) (2022)
- [BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation](https://mlsanity.com/paper/2109.04588/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Moving on from OntoNotes: Coreference Resolution Model Transfer](https://mlsanity.com/paper/2104.08457/) (2021)
- [Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction](https://mlsanity.com/paper/2109.06798/) (2021)
- [Language Model Priming for Cross-Lingual Event Extraction](https://mlsanity.com/paper/2109.12383/) (2021)
- [RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction](https://mlsanity.com/paper/2308.05046/) (2023)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [Domain Generalization with Fourier Transform and Soft Thresholding](https://mlsanity.com/paper/2309.09866/) (2023)
- [Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis](https://mlsanity.com/paper/2411.00691/) (2024)
- [Language Agnostic Code-Mixing Data Augmentation by Predicting Linguistic Patterns](https://mlsanity.com/paper/2211.07628/) (2022)
- [Exploring Data-Efficient Adaptation of Large Language Models for Code Generation](https://mlsanity.com/paper/2403.00046/) (2024)
- [Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning](https://mlsanity.com/paper/2305.19759/) (2023)
- [Multilingual Sentence-Level Semantic Search using Meta-Distillation Learning](https://mlsanity.com/paper/2309.08185/) (2023)
- [Improving Intersession Reproducibility for Forearm Ultrasound based Hand Gesture Classification through an Incremental Learning Approach](https://mlsanity.com/paper/2409.16415/) (2024)
- [Domain Adaptation in 3D Object Detection with Gradual Batch Alternation Training](https://mlsanity.com/paper/2210.10180/) (2022)
- [On Robust Incremental Learning over Many Multilingual Steps](https://mlsanity.com/paper/2210.14307/) (2022)
- [An Augmentation Strategy for Visually Rich Documents](https://mlsanity.com/paper/2212.10047/) (2022)
- [Impact of Subword Pooling Strategy on Cross-lingual Event Detection](https://mlsanity.com/paper/2302.11365/) (2023)
- [Towards Flow Graph Prediction of Open-Domain Procedural Texts](https://mlsanity.com/paper/2305.19497/) (2023)
- [Using Weak Supervision and Data Augmentation in Question Answering](https://mlsanity.com/paper/2309.16175/) (2023)
- [Gradual Fine-Tuning with Graph Routing for Multi-Source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2411.07185/) (2024)
- [Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks](https://mlsanity.com/paper/2610.02549/) (2026)

## Abstract (arXiv)

Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. Such domain adaptation is typically done using one stage of fine-tuning. We demonstrate that gradually fine-tuning in a multi-stage process can yield substantial further gains and can be applied without modifying the model or learning objective.