mlsanityData to 7 Oct 2026

Label Curation Using Agentic AI

arXiv 2602.02564 ↗First posted 30 Jan 2026cs.LG cs.MAcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Data annotation is essential for supervised learning, yet producing accurate, unbiased, and scalable labels remains challenging as datasets grow in size and modality. Traditional human-centric pipelines are costly, slow, and prone to annotator variability, motivating reliability-aware automated annotation. We present AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation), an agentic AI framework for large-scale, multi-modal data annotation. AURA coordinates multiple AI agents to generate and validate labels without requiring ground truth. At its core, AURA adapts a classical probabilistic model that jointly infers latent true labels and annotator reliability via confusion matrices, using Expectation-Maximization to reconcile conflicting annotations and aggregate noisy predictions. Across the four benchmark datasets evaluated, AURA achieves accuracy improvements of up to 5.8% over baseline. In more challenging settings with poor quality annotators, the improvement is up to 50% over baseline. AURA also accurately estimates the reliability of annotators, allowing assessment of annotator quality even without any pre-validation steps.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 29 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

14 of 29 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  9. 2020Are we done with ImageNet?cited by 329
  10. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  11. 2024ARTICLE: Annotator Reliability Through In-Context Learningcited by 6
  12. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation Systemcited by 5
2 more
  1. 2024Data Collection and Labeling Techniques for Machine Learningcited by 4
  2. 2025Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled yet Hard-to-Learn Samples in Noisy Datacited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
  2. 2026Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
  3. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
  4. 2025Adjudicator: Correcting Noisy Labels with a KG-Informed Council of LLM Agents
  5. 2026ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation
  6. 2026Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
  7. 2024MEGAnno+: A Human-LLM Collaborative Annotation System
  8. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents