# Adaptor: Objective-Centric Adaptation Framework for Language Models

arXiv: https://arxiv.org/abs/2203.03989 · HTML: https://mlsanity.com/paper/2203.03989/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models](https://mlsanity.com/paper/2007.06778/) (2020)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://mlsanity.com/paper/1808.04189/) (2018)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Improving QA Generalization by Concurrent Modeling of Multiple Biases](https://mlsanity.com/paper/2010.03338/) (2020)

## Built on by (most cited first, 7 of 7)

- [Resources and Few-shot Learners for In-context Learning in Slavic Languages](https://mlsanity.com/paper/2304.01922/) (2023)
- [Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models](https://mlsanity.com/paper/2305.06841/) (2023)
- [Concept-aware Data Construction Improves In-context Learning of Language Models](https://mlsanity.com/paper/2403.09703/) (2024)
- [Multi-Task Domain Adaptation for Language Grounding with 3D Objects](https://mlsanity.com/paper/2407.02846/) (2024)
- [Soft Alignment Objectives for Robust Adaptation of Language Generation](https://mlsanity.com/paper/2211.16550/) (2022)
- [Concept-aware Training Improves In-context Learning Ability of Language Models](https://mlsanity.com/paper/2305.13775/) (2023)
- [People and Places of Historical Europe: Bootstrapping Annotation Pipeline and a New Corpus of Named Entities in Late Medieval Texts](https://mlsanity.com/paper/2305.16718/) (2023)

## Abstract (arXiv)

Progress in natural language processing research is catalyzed by the possibilities given by the widespread software frameworks. This paper introduces Adaptor library that transposes the traditional model-centric approach composed of pre-training + fine-tuning steps to objective-centric approach, composing the training process by applications of selected objectives. We survey research directions that can benefit from enhanced objective-centric experimentation in multitask training, custom objectives development, dynamic training curricula, or domain adaptation. Adaptor aims to ease reproducibility of these research directions in practice. Finally, we demonstrate the practical applicability of Adaptor in selected unsupervised domain adaptation scenarios.