mlsanityData to 7 Oct 2026

ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking

arXiv 2511.09833 ↗First posted 13 Nov 2025cs.LGcited by 3

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Supervised learning relies on high-quality labeled data, but obtaining such data through human annotation is both expensive and time-consuming. Recent work explores using large language models (LLMs) for annotation, but LLM-generated labels still fall short of human-level quality. To address this problem, we propose the Annotation with Critical Thinking (ACT) data pipeline, where LLMs serve not only as annotators but also as judges to critically identify potential errors. Human effort is then directed towards reviewing only the most "suspicious" cases, significantly improving the human annotation efficiency. Our major contributions are as follows: (1) ACT is applicable to a wide range of domains, including natural language processing (NLP), computer vision (CV), and multimodal understanding, by leveraging multimodal-LLMs (MLLMs). (2) Through empirical studies, we derive 7 insights on how to enhance annotation quality while efficiently reducing the human cost, and then translate these findings into user-friendly guidelines. (3) We theoretically analyze how to modify the loss function so that models trained on ACT data achieve similar performance to those trained on fully human-annotated data. Our experiments show that the performance gap can be reduced to less than 2% on most benchmark datasets while saving up to 90% of human costs.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 65 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

53 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2024A Survey on LLM-as-a-Judgecited by 1,384
38 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  7. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  9. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  12. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  13. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  14. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  15. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  16. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  17. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  18. 2018Learning to Learn from Noisy Labeled Datacited by 160
  19. 2019Deep Self-Learning From Noisy Labelscited by 122
  20. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  21. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  22. 2023PPI++: Efficient Prediction-Powered Inferencecited by 103
  23. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  24. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  25. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  26. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  27. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  28. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  29. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
  30. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  31. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  32. 2024Active Statistical Inferencecited by 46
  33. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  34. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
  35. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  36. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  37. 2024Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichmentcited by 5
  38. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Benchmark Everything Everywhere All at Oncecited by 5
  2. 2026EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Modelscited by 1
  3. 2026Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mappingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Large Language Models as Expert Annotators
  2. 2024HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding
  3. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2026Are Multimodal Large Language Models Good Annotators for Image Tagging?
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  7. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
  8. 2026Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking