# Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset

arXiv: https://arxiv.org/abs/2410.09416 · HTML: https://mlsanity.com/paper/2410.09416/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 11 of 11)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022)
- [Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation](https://mlsanity.com/paper/2112.04554/) (2021)
- [ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models](https://mlsanity.com/paper/2210.04325/) (2022)
- [Consistency and Accuracy of CelebA Attribute Values](https://mlsanity.com/paper/2210.07356/) (2022)
- [Closing the Loop: Testing ChatGPT to Generate Model Explanations to Improve Human Labelling of Sponsored Content on Social Media](https://mlsanity.com/paper/2306.05115/) (2023)
- [On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models](https://mlsanity.com/paper/2108.00114/) (2021)
- [Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)](https://mlsanity.com/paper/2408.00932/) (2024)

## Built on by (most cited first, 5 of 5)

- [TrueSkin: Towards Fair and Accurate Skin Tone Recognition and Generation](https://mlsanity.com/paper/2509.10980/) (2025)
- [On the Robustness of Vision-Language Models in Zero-shot Privacy Classification](https://mlsanity.com/paper/2510.09253/) (2025)
- [Beyond Realism: Learning the Art of Expressive Composition with StickerNet](https://mlsanity.com/paper/2511.20957/) (2025)
- [Which private attributes do VLMs agree on and predict well?](https://mlsanity.com/paper/2602.07931/) (2026)
- [Statistical attribute alignment for black-box generative AI via output post-processing](https://mlsanity.com/paper/2609.31607/) (2026)

## Abstract (arXiv)

This study evaluates the capability of Vision-Language Models (VLMs) in image data annotation by comparing their performance on the CelebA dataset in terms of quality and cost-effectiveness against manual annotation. Annotations from the state-of-the-art LLaVA-NeXT model on 1000 CelebA images are in 79.5% agreement with the original human annotations. Incorporating re-annotations of disagreed cases into a majority vote boosts AI annotation consistency to 89.1% and even higher for more objective labels. Cost assessments demonstrate that AI annotation significantly reduces expenditures compared to traditional manual methods -- representing less than 1% of the costs for manual annotation in the CelebA dataset. These findings support the potential of VLMs as a viable, cost-effective alternative for specific annotation tasks, reducing both financial burden and ethical concerns associated with large-scale manual data annotation. The AI annotations and re-annotations utilized in this study are available on https://github.com/evev2024/EVEV2024_CelebA.