# Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generation

arXiv: https://arxiv.org/abs/2503.11096 · HTML: https://mlsanity.com/paper/2503.11096/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification](https://mlsanity.com/paper/2109.11301/) (2021)
- [Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis](https://mlsanity.com/paper/2309.10771/) (2023)
- [Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis](https://mlsanity.com/paper/2306.17177/) (2023)
- [Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks](https://mlsanity.com/paper/2311.09825/) (2023)
- [LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations](https://mlsanity.com/paper/2412.08580/) (2024)
- [QualiGPT: GPT as an easy-to-use tool for qualitative coding](https://mlsanity.com/paper/2310.07061/) (2023)
- [APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT](https://mlsanity.com/paper/2402.01697/) (2024)
- [Zero-shot Emotion Annotation in Facial Images Using Large Multimodal Models: Benchmarking and Prospects for Multi-Class, Multi-Frame Approaches](https://mlsanity.com/paper/2502.12454/) (2025)
- [VRMN-bD: A Multi-modal Natural Behavior Dataset of Immersive Human Fear Responses in VR Stand-up Interactive Games](https://mlsanity.com/paper/2401.12133/) (2024)

## Built on by (most cited first, 1 of 1)

- [FOR-Prompting: From Objection to Revision via an Asymmetric Prompting Protocol](https://mlsanity.com/paper/2510.01674/) (2025)

## Abstract (arXiv)

Traditional image annotation tasks rely heavily on human effort for object selection and label assignment, making the process time-consuming and prone to decreased efficiency as annotators experience fatigue after extensive work. This paper introduces a novel framework that leverages the visual understanding capabilities of large multimodal models (LMMs), particularly GPT, to assist annotation workflows. In our proposed approach, human annotators focus on selecting objects via bounding boxes, while the LMM autonomously generates relevant labels. This human-AI collaborative framework enhances annotation efficiency by reducing the cognitive and time burden on human annotators. By analyzing the system's performance across various types of annotation tasks, we demonstrate its ability to generalize to tasks such as object recognition, scene description, and fine-grained categorization. Our proposed framework highlights the potential of this approach to redefine annotation workflows, offering a scalable and efficient solution for large-scale data labeling in computer vision. Finally, we discuss how integrating LMMs into the annotation pipeline can advance bidirectional human-AI alignment, as well as the challenges of alleviating the "endless annotation" burden in the face of information overload by shifting some of the work to AI.