mlsanityData to 14 Oct 2026

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

arXiv 2606.27974 ↗First posted 26 Jun 2026cs.CV cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 33 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

32 of 33 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  10. 2023A Survey on Multimodal Large Language Modelscited by 909
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
20 more
  1. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  2. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  3. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  4. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  5. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  6. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  7. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 62
  8. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  9. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  10. 2024Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answeringcited by 20
  11. 2025MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Frameworkcited by 18
  12. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filteringcited by 17
  13. 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrievalcited by 10
  14. 2024CompCap: Improving Multimodal Large Language Models with Composite Captionscited by 8
  15. 2025ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answeringcited by 7
  16. 2025EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Abilitycited by 6
  17. 2026QKVQA: Question-Focused Filtering for Knowledge-based VQAcited by 4
  18. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  19. 2026REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignmentcited by 1
  20. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agentscited by 2
  2. 2026SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytellingcited by 1
  3. 2026Memory-Conditioned Tool Calling for Camera-First Visual Agentscited by 0
  4. 2026LightAIR: Lightweight Action Inversion and Riemannian Rectification for Text-based Person Anomaly Searchcited by 0
  5. 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  2. 2026Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
  3. 2026Efficient Multimodal Planning Agent for Visual Question-Answering
  4. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  5. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  6. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  8. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts