mlsanityData to 7 Oct 2026

Cold-start Active Learning through Self-supervised Language Modeling

arXiv 2010.09535 ↗First posted 19 Oct 2020cs.CL cs.LGcited by 128

Not on the map. Why?

Abstract (arXiv)

Active learning strives to reduce annotation costs by choosing the most critical examples to label. Typically, the active learning strategy is contingent on the classification model. For instance, uncertainty sampling depends on poorly calibrated model confidence scores. In the cold-start setting, active learning is impractical because of model instability and data scarcity. Fortunately, modern NLP provides an additional source of information: pre-trained language models. The pre-training loss can find examples that surprise the model and should be labeled for efficient fine-tuning. Therefore, we treat the language modeling loss as a proxy for classification uncertainty. With BERT, we develop a simple strategy based on the masked language modeling loss that minimizes labeling costs for text classification. Compared to other baselines, our approach reaches higher accuracy within less sampling iterations and computation time.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 27; 2023: 41; 2024: 27; 2025: 15; 2026: 7.

128 papers.

Sort:

By area:

  1. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  2. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  3. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  4. 2022Active Learning on a Budget: Opposite Strategies Suit High and Low Budgetscited by 131
  5. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  6. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  7. 2022Active Learning Through a Covering Lenscited by 62
  8. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  9. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  10. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 53
  11. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  12. 2022A Survey of Active Learning for Natural Language Processingcited by 40
38 more
  1. 2022Active Learning Helps Pretrained Models Learn the Intended Taskcited by 33
  2. 2023A comprehensive survey on deep active learning in medical image analysiscited by 32
  3. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  4. 2021Improving Contrastive Learning on Imbalanced Seed Data via Open-World Samplingcited by 30
  5. 2023Active learning for data streams: a surveycited by 29
  6. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  7. 2022Making Your First Choice: To Address Cold Start Problem in Vision Active Learningcited by 25
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  9. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  10. 2021Diversity Enhanced Active Learning with Strictly Proper Scoring Rulescited by 22
  11. 2022Warm Start Active Learning with Proxy Labels & Selection via Semi-Supervised Fine-Tuningcited by 21
  12. 2024Enhancing Text Classification through LLM-Driven Active Learning and Human Annotationcited by 20
  13. 2021Small-Text: Active Learning for Text Classification in Pythoncited by 18
  14. 2024Optimal Design for Human Preference Elicitationcited by 17
  15. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  16. 2022Exploring Diversity-based Active Learning for 3D Object Detection in Autonomous Drivingcited by 16
  17. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  18. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  19. 2023FreeAL: Towards Human-Free Active Learning in the Era of Large Language Modelscited by 14
  20. 2024Revisiting Active Learning in the Era of Vision Foundation Modelscited by 13
  21. 2023D-CALM: A Dynamic Clustering-based Active Learning Approach for Mitigating Biascited by 12
  22. 2024Evolving Knowledge Distillation with Large Language Models and Active Learningcited by 12
  23. 2021Adapting Coreference Resolution Models through Active Learningcited by 11
  24. 2022Active Few-Shot Learning with FASLcited by 11
  25. 2022Towards Computationally Feasible Deep Active Learningcited by 11
  26. 2023Annotator: A Generic Active Learning Baseline for LiDAR Semantic Segmentationcited by 11
  27. 2024Making Better Use of Unlabelled Data in Bayesian Active Learningcited by 10
  28. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  29. 2023On the Limitations of Simulating Active Learningcited by 9
  30. 2023Towards Balanced Active Learning for Multimodal Classificationcited by 9
  31. 2023COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentationcited by 9
  32. 2022Multilingual Detection of Personal Employment Status on Twittercited by 8
  33. 2023BaSAL: Size-Balanced Warm Start Active Learning for LiDAR Semantic Segmentationcited by 8
  34. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  35. 2024To Label or Not to Label: Hybrid Active Learning for Neural Machine Translationcited by 7
  36. 2024Coherence-Driven Multimodal Safety Dialogue with Active Learning for Embodied Agentscited by 7
  37. 2022Eeny, meeny, miny, moe. How to choose data for morphological inflectioncited by 6
  38. 2023Active Learning for Abstractive Text Summarizationcited by 6

50 most cited of 128 All 128 (JSON)