# Do We Still Need Humans in the Loop? Human vs. LLM Annotation in Active Learning for TikTok Hate Speech Detection

arXiv: https://arxiv.org/abs/2604.13899 · HTML: https://mlsanity.com/paper/2604.13899/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis](https://mlsanity.com/paper/1701.08118/) (2017)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [Practical Obstacles to Deploying Active Learning](https://mlsanity.com/paper/1807.04801/) (2018)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [A Survey of Active Learning for Text Classification using Deep Neural Networks](https://mlsanity.com/paper/2008.07267/) (2020)
- [Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysis](https://mlsanity.com/paper/2308.16705/) (2023)
- [Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation](https://mlsanity.com/paper/2307.01680/) (2023)
- [Hate Personified: Investigating the role of LLMs in content moderation](https://mlsanity.com/paper/2410.02657/) (2024)
- [SubjQA: A Dataset for Subjectivity and Review Comprehension](https://mlsanity.com/paper/2004.14283/) (2020)
- [MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection](https://mlsanity.com/paper/2401.06526/) (2024)
- [A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance](https://mlsanity.com/paper/2506.18576/) (2025)
- [Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates](https://mlsanity.com/paper/2210.04359/) (2022)
- [A Dataset for the Detection of Dehumanizing Language](https://mlsanity.com/paper/2402.08764/) (2024)
- [Reassessing Active Learning Adoption in Contemporary NLP: A Community Survey](https://mlsanity.com/paper/2503.09701/) (2025)
- [Beyond the Explicit: A Bilingual Dataset for Dehumanization Detection in Social Media](https://mlsanity.com/paper/2510.18582/) (2025)

## Abstract (arXiv)

Annotating data remains a costly bottleneck for supervised NLP. Active learning (AL) reduces the number of human labels needed by selecting only the most informative instances, while instruction-tuned LLMs attack the same bottleneck from the other side, making labels cheap enough to annotate entire corpora. This raises two questions: can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both by training supervised hate speech classifiers on a new dataset of 278K German political TikTok comments, comparing human and LLM annotation under matched conditions. LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost, for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-A10B) LLM, and the advantage is robust under soft-label evaluation. It hinges on the annotation interface: only a two-question decomposition mirroring the human annotation task unlocks it. AL provides no reliable advantage over random sampling in our prefiltered pool. Error structure depends on the LLM: only GPT-5.2 matches the human FP/FN balance, while other variants over-flag border-control and economic-competition discourse. Humans remain essential as evaluators; for training labels, the question shifts to which LLM, which interface, and what shape of pool.