mlsanityData to 7 Oct 2026

Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

arXiv 2402.08360 ↗First posted 13 Feb 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 19 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

12 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 10

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  9. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  10. 2022HaGRID - HAnd Gesture Recognition Image Datasetcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  2. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  3. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  4. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 45
  5. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  6. 2025OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learningcited by 24
  7. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  8. 2025UniSVG: A Unified Dataset for Vector Graphic Understanding and Generation with Multimodal Large Language Modelscited by 14
  9. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuningcited by 13
  10. 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Modelscited by 10
  11. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneckcited by 8
  12. 2025R-Genie: Reasoning-Guided Generative Image Editingcited by 6
7 more
  1. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchorscited by 4
  2. 2025Semantic Document Derendering: SVG Reconstruction via Vision-Language Modelingcited by 3
  3. 2026Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuningcited by 3
  4. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMscited by 1
  5. 2026Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMscited by 1
  6. 2026DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddingscited by 0
  7. 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  2. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  3. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  4. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  5. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  6. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  7. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  8. 2025Visual question answering: from early developments to recent advances -- a survey