mlsanityData to 7 Oct 2026

Investigating Multi-source Active Learning for Natural Language Inference

arXiv 2302.06976 ↗First posted 14 Feb 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

In recent years, active learning has been successfully applied to an array of NLP tasks. However, prior work often assumes that training and test data are drawn from the same distribution. This is problematic, as in real-life settings data may stem from several sources of varying relevance and quality. We show that four popular active learning schemes fail to outperform random selection when applied to unlabelled pools comprised of multiple data sources on the task of natural language inference. We reveal that uncertainty-based strategies perform poorly due to the acquisition of collective outliers, i.e., hard-to-learn instances that hamper learning and generalization. When outliers are removed, strategies are found to recover and outperform random baselines. In further analysis, we find that collective outliers vary in form between sources, and show that hard-to-learn data is not always categorically harmful. Lastly, we leverage dataset cartography to introduce difficulty-stratified testing and find that different strategies are affected differently by example learnability and difficulty.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

41 of 61 references.

References · 41

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2017Deep Bayesian Active Learning with Image Datacited by 812
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  8. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  9. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  10. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  11. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
29 more
  1. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  2. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  3. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 148
  4. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  5. 2017Deep Active Learning for Named Entity Recognitioncited by 108
  6. 2019Discriminative Active Learningcited by 97
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  8. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  9. 2019Diverse mini-batch Active Learningcited by 88
  10. 2018Practical Obstacles to Deploying Active Learningcited by 68
  11. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  12. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  13. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  14. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  15. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  16. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  17. 2019Uncertain Natural Language Inferencecited by 35
  18. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  19. 2021Data Shapley Valuation for Efficient Batch Active Learningcited by 27
  20. 2021Cartography Active Learningcited by 26
  21. 2016Active Learning for Speech Recognition: the Power of Gradientscited by 25
  22. 2021Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outlierscited by 22
  23. 2018Active Learning for Interactive Neural Machine Translation of Data Streamscited by 20
  24. 2020Reducing Confusion in Active Learning for Part-Of-Speech Taggingcited by 11
  25. 2021Adapting Coreference Resolution Models through Active Learningcited by 11
  26. 2022Active Learning Over Multiple Domains in Natural Language Taskscited by 11
  27. 2021PowerEvaluationBALD: Efficient Evaluation-Oriented Deep (Bayesian) Active Learning with Stochastic Acquisition Functionscited by 4
  28. 2022Is More Data Better? Re-thinking the Importance of Efficiency in Abusive Language Detection with Transformers-Based Active Learningcited by 4
  29. 2021Active Learning for Argument Strength Estimationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  2. 2023On the Limitations of Simulating Active Learningcited by 9
  3. 2024On the Pros and Cons of Active Learning for Moral Preference Elicitationcited by 6
  4. 2024STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Modelscited by 5
  5. 2023Multi-Domain Learning From Insufficient Annotationscited by 1
  6. 2023Perturbation-Based Two-Stage Multi-Domain Active Learningcited by 1