mlsanityData to 14 Oct 2026

Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset

arXiv 2410.09416 ↗First posted 12 Oct 2024cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This study evaluates the capability of Vision-Language Models (VLMs) in image data annotation by comparing their performance on the CelebA dataset in terms of quality and cost-effectiveness against manual annotation. Annotations from the state-of-the-art LLaVA-NeXT model on 1000 CelebA images are in 79.5% agreement with the original human annotations. Incorporating re-annotations of disagreed cases into a majority vote boosts AI annotation consistency to 89.1% and even higher for more objective labels. Cost assessments demonstrate that AI annotation significantly reduces expenditures compared to traditional manual methods -- representing less than 1% of the costs for manual annotation in the CelebA dataset. These findings support the potential of VLMs as a viable, cost-effective alternative for specific annotation tasks, reducing both financial burden and ethical concerns associated with large-scale manual data annotation. The AI annotations and re-annotations utilized in this study are available on https://github.com/evev2024/EVEV2024_CelebA.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 16 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

11 of 16 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 10

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  4. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  5. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 52
  6. 2022ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Modelscited by 14
  7. 2022Consistency and Accuracy of CelebA Attribute Valuescited by 11
  8. 2023Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Mediacited by 9
  9. 2021On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Modelscited by 6
  10. 2024Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025TrueSkin: Towards Fair and Accurate Skin Tone Recognition and Generationcited by 2
  2. 2025On the Robustness of Vision-Language Models in Zero-shot Privacy Classificationcited by 1
  3. 2025Beyond Realism: Learning the Art of Expressive Composition with StickerNetcited by 0
  4. 2026Which private attributes do VLMs agree on and predict well?cited by 0
  5. 2026Statistical attribute alignment for black-box generative AI via output post-processingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Building and better understanding vision-language models: insights and future directions
  2. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  3. 2026Multilingual Training and Evaluation Resources for Vision-Language Models
  4. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  5. 2026Are Multimodal Large Language Models Good Annotators for Image Tagging?
  6. 2026Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework
  7. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  8. 2024Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost