# The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

arXiv: https://arxiv.org/abs/2411.11758 · HTML: https://mlsanity.com/paper/2411.11758/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizations](https://mlsanity.com/paper/2310.05421/) (2023)
- [Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning](https://mlsanity.com/paper/2109.06860/) (2021)
- [Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language](https://mlsanity.com/paper/2306.16410/) (2023)
- [GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition](https://mlsanity.com/paper/2301.02560/) (2023)
- [Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering](https://mlsanity.com/paper/2306.09996/) (2023)
- [CIC: A Framework for Culturally-Aware Image Captioning](https://mlsanity.com/paper/2402.05374/) (2024)
- [Polarization of Autonomous Generative AI Agents Under Echo Chambers](https://mlsanity.com/paper/2402.12212/) (2024)

## Built on by (most cited first, 9 of 9)

- [When Cultures Meet: Multicultural Text-to-Image Generation](https://mlsanity.com/paper/2502.15972/) (2025)
- [When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation](https://mlsanity.com/paper/2605.16716/) (2026)
- [Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activities](https://mlsanity.com/paper/2504.06313/) (2025)
- [The Curious Case of Curiosity across Human Cultures and LLMs](https://mlsanity.com/paper/2510.12943/) (2025)
- [Beyond Translation: Cross-Cultural Meme Transcreation with Vision-Language Models](https://mlsanity.com/paper/2602.02510/) (2026)
- [Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images](https://mlsanity.com/paper/2604.07338/) (2026)
- [Towards Faithful Sentimental Image Captioning via Evidence-Aware Multi-Agent Reasoning](https://mlsanity.com/paper/2607.25789/) (2026)
- [VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation](https://mlsanity.com/paper/2608.26155/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MosAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adaptable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research. Our dataset and models can be accessed at https://github.com/MichiganNLP/MosAIC.