mlsanityData to 14 Oct 2026

The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

arXiv 2411.11758 ↗First posted 18 Nov 2024cs.CV cs.AI cs.CLcited by 9

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MosAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adaptable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research. Our dataset and models can be accessed at https://github.com/MichiganNLP/MosAIC.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 54 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

25 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CIC: A Framework for Culturally-Aware Image Captioning8 Feb
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering16 Jun

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  9. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  10. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  11. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  12. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
10 more
  1. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  2. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  3. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  4. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  5. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  6. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 61
  7. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  8. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  9. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  10. 2024Polarization of Autonomous Generative AI Agents Under Echo Chamberscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025When Cultures Meet: Multicultural Text-to-Image Generationcited by 7
  2. 2026When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generationcited by 1
  3. 2025Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activitiescited by 0
  4. 2025The Curious Case of Curiosity across Human Cultures and LLMscited by 0
  5. 2026Beyond Translation: Cross-Cultural Meme Transcreation with Vision-Language Modelscited by 0
  6. 2026Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Imagescited by 0
  7. 2026Towards Faithful Sentimental Image Captioning via Evidence-Aware Multi-Agent Reasoningcited by 0
  8. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptationcited by 0
  9. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  2. 2025CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation
  3. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awareness
  4. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts
  5. 2024Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models
  6. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models
  7. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  8. 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation