# Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation

arXiv: https://arxiv.org/abs/2406.12114 · HTML: https://mlsanity.com/paper/2406.12114/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 22 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 6 of 6)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Cold-start Active Learning through Self-supervised Language Modeling](https://mlsanity.com/paper/2010.09535/) (2020)
- [LLMaAA: Making Large Language Models as Active Annotators](https://mlsanity.com/paper/2310.19596/) (2023)
- [Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks](https://mlsanity.com/paper/2311.09825/) (2023)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)

## Built on by (most cited first, 20 of 20)

- [From Selection to Generation: A Survey of LLM-based Active Learning](https://mlsanity.com/paper/2502.11767/) (2025)
- [Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions](https://mlsanity.com/paper/2410.02028/) (2024)
- [Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation](https://mlsanity.com/paper/2506.15068/) (2025)
- [Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Research](https://mlsanity.com/paper/2411.00890/) (2024)
- [GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets](https://mlsanity.com/paper/2504.19898/) (2025)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration](https://mlsanity.com/paper/2505.03985/) (2025)
- [Next Generation Active Learning: Mixture of LLMs in the Loop](https://mlsanity.com/paper/2601.15773/) (2026)
- [Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables](https://mlsanity.com/paper/2502.09073/) (2025)
- [Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements](https://mlsanity.com/paper/2506.10330/) (2025)
- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks](https://mlsanity.com/paper/2603.21404/) (2026)
- [AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions](https://mlsanity.com/paper/2605.13149/) (2026)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)
- [Large Language Models in the Task of Automatic Validation of Text Classifier Predictions](https://mlsanity.com/paper/2505.18688/) (2025)
- [Small sample-based adaptive text classification through iterative and contrastive description refinement](https://mlsanity.com/paper/2508.00957/) (2025)
- [The Impact of Role Design in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2509.23501/) (2025)
- [Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study](https://mlsanity.com/paper/2604.01615/) (2026)
- [A Classifier That Teaches Itself: Self-Improving, Frozen-gate Training (SIFT) for Dynamic Document Classification](https://mlsanity.com/paper/2607.18358/) (2026)
- [DocAnnot -- Accelerating the Creation of Key Information Extraction Datasets with GenAI-Powered Auto-annotation](https://mlsanity.com/paper/2607.24745/) (2026)

## Abstract (arXiv)

In the context of text classification, the financial burden of annotation exercises for creating training data is a critical issue. Active learning techniques, particularly those rooted in uncertainty sampling, offer a cost-effective solution by pinpointing the most instructive samples for manual annotation. Similarly, Large Language Models (LLMs) such as GPT-3.5 provide an alternative for automated annotation but come with concerns regarding their reliability. This study introduces a novel methodology that integrates human annotators and LLMs within an Active Learning framework. We conducted evaluations on three public datasets. IMDB for sentiment analysis, a Fake News dataset for authenticity discernment, and a Movie Genres dataset for multi-label classification.The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels. This strategy achieves an optimal balance between cost efficiency and classification performance. The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.