# ArtAnno: Annotating Implicit Semantics in Artworks through LLM Agent-Driven Bidirectional Human-AI Augmentation

arXiv: https://arxiv.org/abs/2608.05026 · HTML: https://mlsanity.com/paper/2608.05026/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution](https://mlsanity.com/paper/2603.01145/) (2026)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [XSkill: Continual Learning from Experience and Skills in Multimodal Agents](https://mlsanity.com/paper/2603.12056/) (2026)
- [CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent](https://mlsanity.com/paper/2310.06155/) (2023)
- [DeepEdit: Knowledge Editing as Decoding with Constraints](https://mlsanity.com/paper/2401.10471/) (2024)
- [SciSciGPT: Advancing Human-AI Collaboration in the Science of Science](https://mlsanity.com/paper/2504.05559/) (2025)
- [VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework](https://mlsanity.com/paper/2403.09027/) (2024)
- [LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding](https://mlsanity.com/paper/2505.12253/) (2025)
- [Human-LLM Coevolution: Evidence from Academic Writing](https://mlsanity.com/paper/2502.09606/) (2025)
- [Semi-Automatic Data Annotation guided by Feature Space Projection](https://mlsanity.com/paper/2007.13689/) (2020)
- [DEArt: Dataset of European Art](https://mlsanity.com/paper/2211.01226/) (2022)
- [Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use](https://mlsanity.com/paper/2403.02626/) (2024)
- [PosterMate: Audience-driven Collaborative Persona Agents for Poster Design](https://mlsanity.com/paper/2507.18572/) (2025)
- [ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval](https://mlsanity.com/paper/2507.21917/) (2025)
- [CrowdAgent: Multi-Agent Managed Multi-Source Annotation System](https://mlsanity.com/paper/2509.14030/) (2025)
- [VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels](https://mlsanity.com/paper/2507.09008/) (2025)
- [Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration](https://mlsanity.com/paper/2510.26069/) (2025)
- [Knowledge Graph for Intelligent Generation of Artistic Image Creation: Constructing a New Annotation Hierarchy](https://mlsanity.com/paper/2511.03585/) (2025)
- [DuoDrama: Supporting Screenplay Refinement Through LLM-Assisted Human Reflection](https://mlsanity.com/paper/2602.05854/) (2026)
- [Figures as Interfaces: Toward LLM-Native Artifacts for Scientific Discovery](https://mlsanity.com/paper/2604.08491/) (2026)

## Abstract (arXiv)

High-quality annotation of artworks is essential for computational art research, yet extracting implicit semantics remains challenging due to the reliance on culturally grounded meanings and deep contextual knowledge behind the images. Current AI-assisted annotation tools often lack assistance or rely on one-way workflows where experts have to perform extra manual calibrations to improve AI models, resulting in limited efficiency. To address this, we propose Bidirectional Human-AI Augmentation(BiHAA), a closed-loop framework in which skills and domain knowledge base evolve through real-time interaction and bidirectional HAI augmentation. Informed by a formative study with 20 artwork annotators from different backgrounds, we implement this framework in ArtAnno, an artwork annotation system driven by a multi-agent architecture. The system includes a Proactive Agentic Support Module, where AI augments humans through semantic mining and label suggestion, and an Interaction-Driven Evolution Module, where human expertise continuously enhances the AI through distilling annotation trajectories into reusable experience. Evaluation through a user study and two case studies demonstrates that our framework and system improve annotation efficiency, enable knowledge accumulation, and reduce the effort of information seeking and verification for annotators with limited domain expertise. We conclude by discussing broader implications and future directions.