# Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks

arXiv: https://arxiv.org/abs/2402.08360 · HTML: https://mlsanity.com/paper/2402.08360/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 12 of 12)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [LAVIS: A Library for Language-Vision Intelligence](https://mlsanity.com/paper/2209.09019/) (2022)
- [HaGRID - HAnd Gesture Recognition Image Dataset](https://mlsanity.com/paper/2206.08219/) (2022)

## Built on by (most cited first, 20 of 20)

- [Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt](https://mlsanity.com/paper/2406.04031/) (2024)
- [Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention](https://mlsanity.com/paper/2406.12718/) (2024)
- [Reducing Hallucinations in Vision-Language Models via Latent Space Steering](https://mlsanity.com/paper/2410.15778/) (2024)
- [Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift](https://mlsanity.com/paper/2406.18844/) (2024)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)
- [OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning](https://mlsanity.com/paper/2503.16081/) (2025)
- [SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning](https://mlsanity.com/paper/2411.13949/) (2024)
- [UniSVG: A Unified Dataset for Vector Graphic Understanding and Generation with Multimodal Large Language Models](https://mlsanity.com/paper/2508.07766/) (2025)
- [LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning](https://mlsanity.com/paper/2508.06202/) (2025)
- [MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2406.09121/) (2024)
- [VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck](https://mlsanity.com/paper/2601.05547/) (2026)
- [R-Genie: Reasoning-Guided Generative Image Editing](https://mlsanity.com/paper/2505.17768/) (2025)
- [Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors](https://mlsanity.com/paper/2509.21997/) (2025)
- [Semantic Document Derendering: SVG Reconstruction via Vision-Language Modeling](https://mlsanity.com/paper/2511.13478/) (2025)
- [Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2605.26110/) (2026)
- [CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs](https://mlsanity.com/paper/2507.22074/) (2025)
- [Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMs](https://mlsanity.com/paper/2601.16527/) (2026)
- [Enhanced MLLM Black-Box Jailbreaking Attacks and Defenses](https://mlsanity.com/paper/2510.21214/) (2025)
- [DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddings](https://mlsanity.com/paper/2606.22779/) (2026)
- [Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations](https://mlsanity.com/paper/2610.11907/) (2026)

## Abstract (arXiv)

Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.