mlsanityData to 14 Oct 2026

CrowdAgent: Multi-Agent Managed Multi-Source Annotation System

arXiv 2509.14030 ↗First posted 17 Sep 2025cs.AIcited by 5

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

High-quality annotated data is a cornerstone of modern Natural Language Processing (NLP). While recent methods begin to leverage diverse annotation sources-including Large Language Models (LLMs), Small Language Models (SLMs), and human experts-they often focus narrowly on the labeling step itself. A critical gap remains in the holistic process control required to manage these sources dynamically, addressing complex scheduling and quality-cost trade-offs in a unified manner. Inspired by real-world crowdsourcing companies, we introduce CrowdAgent, a multi-agent system that provides end-to-end process control by integrating task assignment, data annotation, and quality/cost management. It implements a novel methodology that rationally assigns tasks, enabling LLMs, SLMs, and human experts to advance synergistically in a collaborative annotation workflow. We demonstrate the effectiveness of CrowdAgent through extensive experiments on six diverse multimodal classification tasks. The source code and video demo are available at https://github.com/QMMMS/CrowdAgent.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 57 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

28 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 27

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2017A Closer Look at Memorization in Deep Networkscited by 942
  5. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  6. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  7. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  8. 2022Is GPT-3 a Good Data Annotator?cited by 208
  9. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  10. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  11. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  12. 2018CrisisMMD: Multimodal Twitter Datasets from Natural Disasterscited by 89
15 more
  1. 2020Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generationcited by 35
  2. 2019Controlled Crowdsourcing for High-Quality QA-SRL Annotationcited by 34
  3. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  4. 2020Analysis of Social Media Data using Multimodal Deep Learning for Disaster Responsecited by 23
  5. 2024Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotationcited by 21
  6. 2024Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotationcited by 16
  7. 2022Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspectivecited by 11
  8. 2018Grounded Textual Entailmentcited by 10
  9. 2024To Label or Not to Label: Hybrid Active Learning for Neural Machine Translationcited by 8
  10. 2024Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4cited by 6
  11. 2025Active Few-Shot Learning for Text Classificationcited by 6
  12. 2024An Active Learning Framework for Inclusive Generation by Large Language Modelscited by 5
  13. 2024Human-LLM Hybrid Text Answer Aggregation for Crowd Annotationscited by 4
  14. 2021Confident in the Crowd: Bayesian Inference to Improve Data Labelling in Crowdsourcingcited by 2
  15. 2023A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correctioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guaranteescited by 1
  2. 2026Label Curation Using Agentic AIcited by 0
  3. 2026Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agentscited by 0
  4. 2026ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentationcited by 0
  5. 2026CALICO: A Human-Centered, Codebook-Aligned System for Annotationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LinguistAgent Technical Report: A Reflective Multi-Model Platform for Automated Linguistic Annotation
  2. 2025Evaluating Large Language Models as Expert Annotators
  3. 2024MEGAnno+: A Human-LLM Collaborative Annotation System
  4. 2025ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies
  5. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
  6. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
  7. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
  8. 2023TrainerAgent: Customizable and Efficient Model Training through LLM-Powered Multi-Agent System