mlsanityData to 7 Oct 2026

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

arXiv 2310.15638 ↗First posted 24 Oct 2023cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

33 of 53 references.

References · 33

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  11. 2022Is GPT-3 a Good Data Annotator?cited by 208
  12. 2023ChatGPT: Jack of all trades, master of nonecited by 204
21 more
  1. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  2. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  3. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  4. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  5. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  6. 2018Conversations Gone Awry: Detecting Early Signs of Conversational Failurecited by 73
  7. 2018Weakly-Supervised Neural Text Classificationcited by 66
  8. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  9. 2020Learning from Rules Generalizing Labeled Exemplarscited by 59
  10. 2020Generating Counter Narratives against Online Hate Speech: Data and Strategiescited by 58
  11. 2023Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remarkcited by 54
  12. 2018Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scalecited by 52
  13. 2020Named Entity Recognition without Labelled Data: A Weak Supervision Approachcited by 46
  14. 2023Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPTcited by 41
  15. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
  16. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  17. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  18. 2021SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasetscited by 18
  19. 2022Fast Rates in Pool-Based Batch Active Learningcited by 15
  20. 2021Creating Training Sets via Weak Indirect Supervisioncited by 14
  21. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 34; 2026: 17.

93 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  3. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  4. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  5. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  6. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  7. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  8. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  9. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  10. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  11. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  12. 2024Evaluating Large Language Models with Psychometricscited by 22
38 more
  1. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 20
  4. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  5. 2024Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselorscited by 16
  6. 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 16
  7. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  8. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  9. 2025AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depressioncited by 13
  10. 2025Mapping Caregiver Needs to AI Chatbot Design: Strengths and Gaps in Mental Health Support for Alzheimer's and Dementia Caregiverscited by 13
  11. 2024HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agentcited by 12
  12. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  13. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  14. 2024GPTs Are Multilingual Annotators for Sequence Generation Taskscited by 11
  15. 2024Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructionscited by 11
  16. 2025Cascaded Language Models for Cost-effective Human-AI Decision-Makingcited by 11
  17. 2024ARL2: Aligning Retrievers for Black-box Large Language Models via Self-guided Adaptive Relevance Labelingcited by 10
  18. 2024Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Futurecited by 10
  19. 2024CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignmentcited by 10
  20. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  21. 2024Decoding the Narratives: Analyzing Personal Drug Experiences Shared on Redditcited by 9
  22. 2025Few-shot LLM Synthetic Data with Distribution Matchingcited by 9
  23. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  24. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 9
  25. 2024LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?cited by 8
  26. 2024Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistlescited by 7
  27. 2025Probably Approximately Correct Labelscited by 7
  28. 2024Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Modelcited by 6
  29. 2024OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistantscited by 6
  30. 2024Language Model-Driven Data Pruning Enables Efficient Active Learningcited by 6
  31. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6
  32. 2025Evaluating how LLM annotations represent diverse views on contentious topicscited by 6
  33. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  34. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation Systemcited by 5
  35. 2025Robust Sampling for Active Statistical Inferencecited by 5
  36. 2024Can Active Label Correction Improve LLM-based Modular AI Systems?cited by 4
  37. 2024Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptationcited by 4
  38. 2024Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Modelscited by 4

50 most cited of 93 All 93 (JSON)