mlsanityData to 7 Oct 2026

Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

arXiv 2412.18351 ↗First posted 24 Dec 2024cs.CL cs.AIcited by 3

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 41 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

33 of 41 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  11. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  12. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
21 more
  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  3. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  4. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  5. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  6. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  7. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  8. 2017Natural Language Processing: State of The Art, Current Trends and Challengescited by 114
  9. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  10. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  11. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  12. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  13. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  14. 2020Deep Multimodal Neural Architecture Searchcited by 34
  15. 2023Agent Instructs Large Language Models to be General Zero-Shot Reasonerscited by 32
  16. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  17. 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
  18. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  19. 2022A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQAcited by 18
  20. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
  21. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQAcited by 4
  2. 2025VIKSER: Visual Knowledge-Driven Self-Reinforcing Reasoning Frameworkcited by 1
  3. 2025Zero-Shot Anomaly Detection in Battery Thermal Images Using Visual Question Answering with Prior Knowledgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering
  2. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
  3. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  4. 2025A Survey of Large Language Model Agents for Question Answering
  5. 2026Understanding Multi-Agent Reasoning with Large Language Models for Cartoon VQA
  6. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
  7. 2024Find The Gap: Knowledge Base Reasoning For Visual Question Answering
  8. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering