mlsanityData to 14 Oct 2026

MEGAnno+: A Human-LLM Collaborative Annotation System

arXiv 2402.18050 ↗First posted 28 Feb 2024cs.CL cs.HCcited by 47

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

16 of 31 references.

References · 16

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  7. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  8. 2023How is ChatGPT's behavior changing over time?cited by 270
  9. 2022Is GPT-3 a Good Data Annotator?cited by 208
  10. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  11. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  12. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
4 more
  1. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  2. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  3. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  4. 2022Label Sleuth: From Unlabeled Text to a Classifier in a Few Hourscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 20; 2026: 15.

47 papers.

Sort:

By area:

  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  2. 2024Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertisecited by 50
  3. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  4. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 42
  5. 2025Stay Focused: Problem Drift in Multi-Agent Debatecited by 23
  6. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  7. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  8. 2025Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Modelscited by 15
  9. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 10
  10. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignmentcited by 9
  11. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  12. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
35 more
  1. 2025Position: LLMs Can be Good Tutors in English Educationcited by 6
  2. 2024Fake News Detection: Comparative Evaluation of BERT-like Models and Large Language Models with Generative AI-Annotated Datacited by 5
  3. 2025From Conversation to Automation: Leveraging LLMs for Problem-Solving Therapy Analysiscited by 4
  4. 2025Can Third-parties Read Our Emotions?cited by 4
  5. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  6. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinkingcited by 4
  7. 2024BEADs: Bias Evaluation Across Domainscited by 3
  8. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3
  9. 2024Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?cited by 3
  10. 2025Co-DETECT: Collaborative Discovery of Edge Cases in Text Classificationcited by 3
  11. 2025eSkinHealth: A Multimodal Dataset for Neglected Tropical Skin Diseasescited by 3
  12. 2026Are Multimodal Large Language Models Good Annotators for Image Tagging?cited by 3
  13. 2024Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Frameworkcited by 2
  14. 2026Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfercited by 2
  15. 2024Scaling Technology Acceptance Analysis with Large Language Model (LLM) Annotation Systemscited by 1
  16. 2025Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extractioncited by 1
  17. 2025Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?cited by 1
  18. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  19. 2026ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarizationcited by 1
  20. 2024Investigating Low-Cost LLM Annotation for~Spoken Dialogue Understanding Datasetscited by 0
  21. 2024An Empirical Study of Validating Synthetic Data for Formula Generationcited by 0
  22. 2025Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yetcited by 0
  23. 2025CoLa: Learning to Interactively Collaborate with Large Language Modelscited by 0
  24. 2025Co-Authoring the Self: A Human-AI Interface for Interest Reflection in Recommenderscited by 0
  25. 2026Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluationcited by 0
  26. 2026AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestionscited by 0
  27. 2026Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotationcited by 0
  28. 2026WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classificationcited by 0
  29. 2026Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videoscited by 0
  30. 2026Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotationcited by 0
  31. 2026Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agentscited by 0
  32. 2026VLMs for Videogame Data Annotationcited by 0
  33. 2026CALICO: A Human-Centered, Codebook-Aligned System for Annotationcited by 0
  34. 2026Structured Claim-Level Discourse Representations for Dense Health Narrativescited by 0
  35. 2026JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
  2. 2023Aligning Large Language Models with Human: A Survey
  3. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
  4. 2025Users as Annotators: LLM Preference Learning from Comparison Mode
  5. 2025Evaluating Large Language Models as Expert Annotators
  6. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels
  7. 2025Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
  8. 2024DuetML: Human-LLM Collaborative Machine Learning Framework for Non-Expert Users