On Dataset Transferability in Active Learning for Transformers
Not on the map. Why?
Abstract (arXiv)
Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 16 May 2023.
Builds on
18 of 28 references.
References · 18
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2018Neural Network Acceptability Judgmentscited by 958
- 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
- 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
- 2019Diverse mini-batch Active Learningcited by 89
- 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 75
- 2018Practical Obstacles to Deploying Active Learningcited by 68
- 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 60
- 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
6 more
- 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
- 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 32
- 2020Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learningcited by 25
- 2021Rissanen Data Analysis: Examining Dataset Characteristics via Description Lengthcited by 22
- 2022Towards Computationally Feasible Deep Active Learningcited by 11
- 2022To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Modelscited by 3
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.