mlsanityData to 14 Oct 2026

Question Aware Vision Transformer for Multimodal Reasoning

arXiv 2402.05472 ↗First posted 8 Feb 2024cs.CVcited by 32

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 54 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

25 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
12 more
  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  3. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  4. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  5. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  6. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  7. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  8. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  9. 2022Multimodal Semi-Supervised Learning for Text Recognitioncited by 16
  10. 2023CLIPTER: Looking at the Bigger Picture in Scene Text Recognitioncited by 16
  11. 2023Towards Models that Can See and Readcited by 15
  12. 2023CLIPAG: Towards Generator-Free Text-to-Image Generationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 11; 2026: 6.

32 papers.

Sort:

By area:

  1. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 107
  2. 2024Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuningcited by 59
  3. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  4. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  5. 2024DocVLM: Make Your VLM an Efficient Readercited by 28
  6. 2024Pathways on the Image Manifold: Image Editing via Video Generationcited by 27
  7. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  8. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 13
  9. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12
  10. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  11. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
  12. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 7
20 more
  1. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Drivingcited by 6
  2. 2024NL-Eye: Abductive NLI for Imagescited by 5
  3. 2024InterCLIP-MEP: Interactive CLIP and Memory-Enhanced Predictor for Multi-modal Sarcasm Detectioncited by 4
  4. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  5. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQAcited by 4
  6. 2025TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusingcited by 4
  7. 2024Text-to-Image Generation Via Energy-Based CLIPcited by 3
  8. 2025From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoningcited by 3
  9. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  10. 2025Video-VoT-R1: An efficient video inference model integrating image packing and AoE architecturecited by 2
  11. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  12. 2026DODO: Discrete OCR Diffusion Modelscited by 2
  13. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  14. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Modelscited by 1
  15. 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Predictioncited by 1
  16. 2025BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQAcited by 0
  17. 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understandingcited by 0
  18. 2026iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understandingcited by 0
  19. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Modelscited by 0
  20. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MV-CoRe: Multimodal Visual-Conceptual Reasoning for Complex Visual Question Answering
  2. 2026ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering
  3. 2025Instruction-tuned Self-Questioning Framework for Multimodal Reasoning
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  6. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation
  7. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  8. 2025VLMT: Vision-Language Multimodal Transformer for Multimodal Multi-hop Question Answering