mlsanityData to 14 Oct 2026

Multi-task Active Learning for Pre-trained Transformer-based Models

arXiv 2208.05379 ↗First posted 10 Aug 2022cs.CL cs.HC cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Multi-task learning, in which several tasks are jointly learned by a single model, allows NLP models to share information from multiple annotations and may facilitate better predictions when the tasks are inter-related. This technique, however, requires annotating the same text with multiple annotation schemes which may be costly and laborious. Active learning (AL) has been demonstrated to optimize annotation processes by iteratively selecting unlabeled examples whose annotation is most valuable for the NLP model. Yet, multi-task active learning (MT-AL) has not been applied to state-of-the-art pre-trained Transformer-based NLP models. This paper aims to close this gap. We explore various multi-task selection criteria in three realistic multi-task scenarios, reflecting different relations between the participating tasks, and demonstrate the effectiveness of multi-task compared to single-task selection. Our results suggest that MT-AL can be effectively used in order to minimize annotation efforts for multi-task NLP models.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 68 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

35 of 68 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  12. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
23 more
  1. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  2. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  3. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  4. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  5. 2020Calibration of Pre-trained Transformerscited by 315
  6. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  7. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  8. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  9. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  10. 2019Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extractioncited by 73
  11. 2018A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Taskscited by 54
  12. 2020Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learningcited by 25
  13. 2019Deep Contextualized Self-training for Low Resource Dependency Parsingcited by 24
  14. 2018Active Learning for Interactive Neural Machine Translation of Data Streamscited by 21
  15. 2021Weakly Supervised Named Entity Tagging with Learnable Logical Rulescited by 21
  16. 2018Comprehensive Supersense Disambiguation of English Prepositions and Possessivescited by 20
  17. 2018An Annotated Corpus for Machine Reading of Instructions in Wet Lab Protocolscited by 18
  18. 2020Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Datacited by 17
  19. 2021PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsingcited by 13
  20. 2020Active Learning for Coreference Resolution using Discrete Annotationcited by 12
  21. 2016Encoder-decoder with Focus-mechanism for Sequence Labelling Based Spoken Language Understandingcited by 9
  22. 2020Pre-trained Language Model Based Active Learning for Sentence Matchingcited by 7
  23. 2018Multi-Task Active Learning for Neural Semantic Role Labeling on Low Resource Conversational Corpuscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 6; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  2. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  3. 2022MEAL: Stable and Active Learning for Few-Shot Promptingcited by 20
  4. 2023D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Biascited by 12
  5. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  6. 2024Active Fine-Tuning of Multi-Task Policiescited by 9
  7. 2024An Active Learning Framework for Inclusive Generation by Large Language Modelscited by 5
  8. 2023Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Trainingcited by 2
  9. 2024GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noisecited by 2
  10. 2023EASE: An Easily-Customized Annotation System Powered by Efficiency Enhancement Mechanismscited by 1
  11. 2023Semi-Automated Construction of Food Composition Knowledge Basecited by 0
  12. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
4 more
  1. 2024Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Reviewcited by 0
  2. 2025ALPET: Active Few-shot Learning for Citation Worthiness Detection in Low-Resource Wikipedia Languagescited by 0
  3. 2025Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noisecited by 0
  4. 2026When Is Coarse Supervision Worth It? Cost-Aware Learning under Unknown Aggregationcited by 0