mlsanityData to 7 Oct 2026

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

arXiv 2411.17558 ↗First posted 26 Nov 2024cs.CL cs.CVcited by 74

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 270 of its 495 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

270 of 495 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  2. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise19 Dec
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  9. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  10. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 260

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  12. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
38 more
  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  13. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  14. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  15. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  16. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  17. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  18. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  19. 2015VQA: Visual Question Answeringcited by 2,489
  20. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  21. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  22. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  23. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  24. 2019Towards VQA Models That Can Readcited by 1,696
  25. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  26. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  27. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  28. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  29. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  30. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  31. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  32. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  33. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  34. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  35. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  36. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  37. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  38. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173

50 most cited of 260 All 260 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 38; 2026: 35.

74 papers.

Sort:

By area:

  1. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  2. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  4. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  6. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  7. 2025DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokenscited by 21
  8. 2025Refine Knowledge of Large Language Models via Adaptive Contrastive Learningcited by 14
  9. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  10. 2025Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoningcited by 13
  11. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12
  12. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarkscited by 10
38 more
  1. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  2. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  3. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  4. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 8
  5. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  6. 2026EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configurationcited by 7
  7. 2026On the Generalization Capacities of MLLMs for Spatial Intelligencecited by 7
  8. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  9. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  10. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Studycited by 5
  11. 2025DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?cited by 5
  12. 2026Test-Time Attention Purification for Backdoored Large Vision Language Modelscited by 5
  13. 2025From Token to Line: Enhancing Code Generation with a Long-Term Perspectivecited by 4
  14. 2025GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Modelscited by 4
  15. 2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
  16. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Modelscited by 4
  17. 2025MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuningcited by 3
  18. 2025Reimagining Urban Science: Scaling Causal Inference with Large Language Modelscited by 3
  19. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  20. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  21. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  22. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 3
  23. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  24. 2025From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationshipcited by 2
  25. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2
  26. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  27. 2026Spectral Heat Flow for Conservative Token Condensation in Vision-Language Modelscited by 2
  28. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 2
  29. 2025Harnessing Adaptive Topology Representations for Zero-Shot Graph Question Answeringcited by 1
  30. 2025BLaVe-CoT: Consistency-Aware Visual Question Answering for Blind and Low Vision Userscited by 1
  31. 2025Fast Thinking for Large Language Modelscited by 1
  32. 2025I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMscited by 1
  33. 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Memecited by 1
  34. 2026Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignmentcited by 1
  35. 2026Universal Adversarial Attacks against Closed-Source MLLMs via Target-View Routed Meta Optimizationcited by 1
  36. 2026VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defensecited by 1
  37. 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
  38. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  2. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  3. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  4. 2025Visual question answering: from early developments to recent advances -- a survey
  5. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
  7. 2023Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA
  8. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions