mlsanityData to 7 Oct 2026

Revisiting Active Learning under (Human) Label Variation

arXiv 2507.02593 ↗First posted 3 Jul 2025cs.CL cs.HC cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Access to high-quality labeled data remains a limiting factor in applied supervised learning. While label variation (LV), i.e., differing labels for the same instance, is common, especially in natural language processing, annotation frameworks often still rest on the assumption of a single ground truth. This overlooks human label variation (HLV), the occurrence of plausible differences in annotations, as an informative signal. Similarly, active learning (AL), a popular approach to optimizing the use of limited annotation budgets in training ML models, often relies on at least one of several simplifying assumptions, which rarely hold in practice when acknowledging HLV. In this paper, we examine foundational assumptions about truth and label nature, highlighting the need to decompose observed LV into signal (e.g., HLV) and noise (e.g., annotation error). We survey how the AL and (H)LV communities have addressed -- or neglected -- these distinctions and propose a conceptual framework for incorporating HLV throughout the AL loop, including instance selection, annotator choice, and label representation. We further discuss the integration of large language models (LLM) as annotators. Our work aims to lay a conceptual foundation for HLV-aware active learning, better reflecting the complexities of real-world annotation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 80 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

28 of 80 references.

References · 28

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  3. 2019Human uncertainty makes classification more robustcited by 231
  4. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  5. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  6. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  7. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 124
  8. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  9. 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 65
  10. 2020Disentangling Human Error from the Ground Truth in Segmentation of Medical Imagescited by 58
  11. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  12. 2017Active Learning for Cost-Sensitive Classificationcited by 40
16 more
  1. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  2. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  3. 2023Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' Viewcited by 27
  4. 2019Fine-Grained Argument Unit Recognition and Classificationcited by 26
  5. 2015Active Learning from Weak and Strong Labelerscited by 22
  6. 2024Annotator-Centric Active Learning for Subjective NLP Taskscited by 18
  7. 2023ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variationcited by 17
  8. 2019A Crowdsourced Frame Disambiguation Corpus with Ambiguitycited by 16
  9. 2023Ecologically Valid Explanations for Label Variation in NLIcited by 16
  10. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 16
  11. 2023ActiveGLAE: A Benchmark for Deep Active Learning with Transformerscited by 14
  12. 2016Active Learning from Imperfect Labelerscited by 13
  13. 2021Cost-Accuracy Aware Adaptive Labeling for Active Learningcited by 13
  14. 2025Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistancecited by 11
  15. 2021Clean or Annotate: How to Spend a Limited Data Collection Budgetcited by 10
  16. 2023Human-in-the-loop: Towards Label Embeddings for Measuring Classification Difficultycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gapscited by 4
  2. 2026Structured Credal Learningcited by 1
  3. 2026The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Predictioncited by 0
  4. 2026From Parasocial Scripts to Dyadic Persistence in Autonomous AI-Agent Communitiescited by 0