# Sampling Bias in Deep Active Classification: An Empirical Study

arXiv: https://arxiv.org/abs/1909.09389 · HTML: https://mlsanity.com/paper/1909.09389/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 30 of 36)

- [Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs](https://mlsanity.com/paper/2305.11860/) (2023)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [On the Importance of Effectively Adapting Pretrained Language Models for Active Learning](https://mlsanity.com/paper/2104.08320/) (2021)
- [Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding](https://mlsanity.com/paper/2312.05328/) (2023)
- [MEAL: Stable and Active Learning for Few-Shot Prompting](https://mlsanity.com/paper/2211.08358/) (2022)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models](https://mlsanity.com/paper/2311.15614/) (2023)
- [More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering](https://mlsanity.com/paper/2311.09782/) (2023)
- [Multi-class Text Classification using BERT-based Active Learning](https://mlsanity.com/paper/2104.14289/) (2021)
- [Towards Computationally Feasible Deep Active Learning](https://mlsanity.com/paper/2205.03598/) (2022)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [An Information Retrieval Approach to Building Datasets for Hate Speech Detection](https://mlsanity.com/paper/2106.09775/) (2021)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)
- [Bayesian Active Learning for Sim-to-Real Robotic Perception](https://mlsanity.com/paper/2109.11547/) (2021)
- [Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias](https://mlsanity.com/paper/2205.13698/) (2022)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Investigating Multi-source Active Learning for Natural Language Inference](https://mlsanity.com/paper/2302.06976/) (2023)
- [Deep Active Learning for Text Classification with Diverse Interpretations](https://mlsanity.com/paper/2108.10687/) (2021)
- [TrustAL: Trustworthy Active Learning using Knowledge Distillation](https://mlsanity.com/paper/2201.11661/) (2022)
- [From Categories to Classifiers: Name-Only Continual Learning by Exploring the Web](https://mlsanity.com/paper/2311.11293/) (2023)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis](https://mlsanity.com/paper/2212.11680/) (2022)
- [On Dataset Transferability in Active Learning for Transformers](https://mlsanity.com/paper/2305.09807/) (2023)
- [Active Learning for Natural Language Generation](https://mlsanity.com/paper/2305.15040/) (2023)
- [Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models](https://mlsanity.com/paper/2112.02542/) (2021)
- [On the Fragility of Active Learners for Text Classification](https://mlsanity.com/paper/2403.15744/) (2024)
- [CNN-Trans-Enc: A CNN-Enhanced Transformer-Encoder On Top Of Static BERT representations for Document Classification](https://mlsanity.com/paper/2209.06344/) (2022)
- [ALANNO: An Active Learning Annotation System for Mortals](https://mlsanity.com/paper/2211.06224/) (2022)
- [Clustering Algorithms and RAG Enhancing Semi-Supervised Text Classification with Large LLMs](https://mlsanity.com/paper/2411.06175/) (2024)

## Abstract (arXiv)

The exploding cost and time needed for data labeling and model training are bottlenecks for training DNN models on large datasets. Identifying smaller representative data samples with strategies like active learning can help mitigate such bottlenecks. Previous works on active learning in NLP identify the problem of sampling bias in the samples acquired by uncertainty-based querying and develop costly approaches to address it. Using a large empirical study, we demonstrate that active set selection using the posterior entropy of deep models like FastText.zip (FTZ) is robust to sampling biases and to various algorithmic choices (query size and strategies) unlike that suggested by traditional literature. We also show that FTZ based query strategy produces sample sets similar to those from more sophisticated approaches (e.g ensemble networks). Finally, we show the effectiveness of the selected samples by creating tiny high-quality datasets, and utilizing them for fast and cheap training of large models. Based on the above, we propose a simple baseline for deep active text classification that outperforms the state-of-the-art. We expect the presented work to be useful and informative for dataset compression and for problems involving active, semi-supervised or online learning scenarios. Code and models are available at: https://github.com/drimpossible/Sampling-Bias-Active-Learning