# Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2406.09206 · HTML: https://mlsanity.com/paper/2406.09206/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 66 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Active Learning by Acquiring Contrastive Examples](https://mlsanity.com/paper/2109.03764/) (2021)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Cold-start Active Learning through Self-supervised Language Modeling](https://mlsanity.com/paper/2010.09535/) (2020)
- [Practical Obstacles to Deploying Active Learning](https://mlsanity.com/paper/1807.04801/) (2018)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [Rethinking deep active learning: Using unlabeled data at model training](https://mlsanity.com/paper/1911.08177/) (2019)
- [Sampling Bias in Deep Active Classification: An Empirical Study](https://mlsanity.com/paper/1909.09389/) (2019)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [On the Importance of Effectively Adapting Pretrained Language Models for Active Learning](https://mlsanity.com/paper/2104.08320/) (2021)
- [Active Learning with Partial Feedback](https://mlsanity.com/paper/1802.07427/) (2018)
- [Cartography Active Learning](https://mlsanity.com/paper/2109.04282/) (2021)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval](https://mlsanity.com/paper/2305.10703/) (2023)
- [Small-Text: Active Learning for Text Classification in Python](https://mlsanity.com/paper/2107.10314/) (2021)
- [Neighborhood-Regularized Self-Training for Learning with Few Labels](https://mlsanity.com/paper/2301.03726/) (2023)
- [Multilingual Detection of Personal Employment Status on Twitter](https://mlsanity.com/paper/2203.09178/) (2022)
- [Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Training](https://mlsanity.com/paper/2208.08749/) (2022)
- [SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training](https://mlsanity.com/paper/2210.12653/) (2022)

## Abstract (arXiv)

Active learning is an iterative labeling process that is used to obtain a small labeled subset, despite the absence of labeled data, thereby enabling to train a model for supervised tasks such as text classification. While active learning has made considerable progress in recent years due to improvements provided by pre-trained language models, there is untapped potential in the often neglected unlabeled portion of the data, although it is available in considerably larger quantities than the usually small set of labeled data. In this work, we investigate how self-training, a semi-supervised approach that uses a model to obtain pseudo-labels for unlabeled data, can be used to improve the efficiency of active learning for text classification. Building on a comprehensive reproduction of four previous self-training approaches, some of which are evaluated for the first time in the context of active learning or natural language processing, we introduce HAST, a new and effective self-training strategy, which is evaluated on four text classification benchmarks. Our results show that it outperforms the reproduced self-training approaches and reaches classification results comparable to previous experiments for three out of four datasets, using as little as 25% of the data. The code is publicly available at https://github.com/chschroeder/self-training-for-sample-efficient-active-learning .