mlsanityData to 7 Oct 2026

Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models

arXiv 2406.09206 ↗First posted 13 Jun 2024cs.CL cs.AI cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

Active learning is an iterative labeling process that is used to obtain a small labeled subset, despite the absence of labeled data, thereby enabling to train a model for supervised tasks such as text classification. While active learning has made considerable progress in recent years due to improvements provided by pre-trained language models, there is untapped potential in the often neglected unlabeled portion of the data, although it is available in considerably larger quantities than the usually small set of labeled data. In this work, we investigate how self-training, a semi-supervised approach that uses a model to obtain pseudo-labels for unlabeled data, can be used to improve the efficiency of active learning for text classification. Building on a comprehensive reproduction of four previous self-training approaches, some of which are evaluated for the first time in the context of active learning or natural language processing, we introduce HAST, a new and effective self-training strategy, which is evaluated on four text classification benchmarks. Our results show that it outperforms the reproduced self-training approaches and reaches classification results comparable to previous experiments for three out of four datasets, using as little as 25% of the data. The code is publicly available at https://github.com/chschroeder/self-training-for-sample-efficient-active-learning .

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 27 of its 66 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

27 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Zero-Shot Text Classification with Self-Training31 Oct
  2. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learning13 Sep

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  8. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  9. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  10. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  11. 2022Efficient Few-Shot Learning Without Promptscited by 135
  12. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
13 more
  1. 2018Practical Obstacles to Deploying Active Learningcited by 68
  2. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  3. 2019Rethinking deep active learning: Using unlabeled data at model trainingcited by 39
  4. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  5. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  6. 2018Active Learning with Partial Feedbackcited by 27
  7. 2021Cartography Active Learningcited by 26
  8. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  9. 2021Small-Text: Active Learning for Text Classification in Pythoncited by 18
  10. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  11. 2022Multilingual Detection of Personal Employment Status on Twittercited by 8
  12. 2022Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Trainingcited by 5
  13. 2022SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Trainingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Sets
  2. 2021AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models
  3. 2021Revisiting Self-Training for Few-Shot Learning of Language Model
  4. 2024Self-supervised Pre-training of Text Recognizers
  5. 2024Improving Classification Performance With Human Feedback: Label a few, we label the rest
  6. 2018Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
  7. 2023KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generation
  8. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator