# ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarios

arXiv: https://arxiv.org/abs/2405.10808 · HTML: https://mlsanity.com/paper/2405.10808/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 17 May 2024.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Cold-start Active Learning through Self-supervised Language Modeling](https://mlsanity.com/paper/2010.09535/) (2020)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [Improving and Simplifying Pattern Exploiting Training](https://mlsanity.com/paper/2103.11955/) (2021)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Practical Obstacles to Deploying Active Learning](https://mlsanity.com/paper/1807.04801/) (2018)
- [Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers](https://mlsanity.com/paper/2107.05687/) (2021)
- [PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models](https://mlsanity.com/paper/2204.01172/) (2022)
- [A Survey of Active Learning for Natural Language Processing](https://mlsanity.com/paper/2210.10109/) (2022)
- [Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates](https://mlsanity.com/paper/2101.08133/) (2021)
- [Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning](https://mlsanity.com/paper/2012.02462/) (2020)
- [Small-Text: Active Learning for Text Classification in Python](https://mlsanity.com/paper/2107.10314/) (2021)

## Built on by (most cited first, 18 of 18)

- [From Selection to Generation: A Survey of LLM-based Active Learning](https://mlsanity.com/paper/2502.11767/) (2025)
- [Active Learning for Direct Preference Optimization](https://mlsanity.com/paper/2503.01076/) (2025)
- [ProbeLLM: Automating Principled Diagnosis of LLM Failures](https://mlsanity.com/paper/2602.12966/) (2026)
- [Position: Enough of Scaling LLMs! Lets Focus on Downscaling](https://mlsanity.com/paper/2505.00985/) (2025)
- [Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided Strategy Selection](https://mlsanity.com/paper/2412.00353/) (2024)
- [Training-Free Active Learning Framework in Materials Science with Large Language Models](https://mlsanity.com/paper/2511.19730/) (2025)
- [Active Prompt Learning with Vision-Language Model Priors](https://mlsanity.com/paper/2411.16722/) (2024)
- [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025)
- [Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR](https://mlsanity.com/paper/2601.22595/) (2026)
- [Inhibitory Attacks on Backdoor-based Fingerprinting for Large Language Models](https://mlsanity.com/paper/2601.04261/) (2026)
- [AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions](https://mlsanity.com/paper/2605.13149/) (2026)
- [From Human Annotation to Automation: LLM-in-the-Loop Active Learning for Arabic Sentiment Analysis](https://mlsanity.com/paper/2509.23515/) (2025)
- [Efficient Deep Learning for Biometrics: Overview, Challenges and Trends in Ear of Frugal AI](https://mlsanity.com/paper/2602.08809/) (2026)
- [Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognition](https://mlsanity.com/paper/2602.19385/) (2026)
- [Efficient Exploration at Scale](https://mlsanity.com/paper/2603.17378/) (2026)
- [Activation-Based Active Learning for In-Context Learning: Challenges and Insights](https://mlsanity.com/paper/2606.05134/) (2026)
- [LLMs as Acquisition Policies for Finite-Pool Materials Optimization: A Controlled Study](https://mlsanity.com/paper/2608.19790/) (2026)
- [Unsupervised Transfer Clustering for Mitigating Cold Start in Active Prompt Learning](https://mlsanity.com/paper/2609.05636/) (2026)

## Abstract (arXiv)

Active learning is designed to minimize annotation efforts by prioritizing instances that most enhance learning. However, many active learning strategies struggle with a `cold-start' problem, needing substantial initial data to be effective. This limitation reduces their utility in the increasingly relevant few-shot scenarios, where the instance selection has a substantial impact. To address this, we introduce ActiveLLM, a novel active learning approach that leverages Large Language Models such as GPT-4, o1, Llama 3, or Mistral Large for selecting instances. We demonstrate that ActiveLLM significantly enhances the classification performance of BERT classifiers in few-shot scenarios, outperforming traditional active learning methods as well as improving the few-shot learning methods ADAPET, PERFECT, and SetFit. Additionally, ActiveLLM can be extended to non-few-shot scenarios, allowing for iterative selections. In this way, ActiveLLM can even help other active learning strategies to overcome their cold-start problem. Our results suggest that ActiveLLM offers a promising solution for improving model performance across various learning setups.