mlsanityData to 14 Oct 2026

X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning

arXiv 2311.18799 ↗First posted 30 Nov 2023cs.CV cs.CLcited by 80

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Recent research has achieved significant advancements in visual reasoning tasks through learning image-to-language projections and leveraging the impressive reasoning abilities of Large Language Models (LLMs). This paper introduces an efficient and effective framework that integrates multiple modalities (images, 3D, audio and video) to a frozen LLM and demonstrates an emergent ability for cross-modal reasoning (2+ modality inputs). Our approach explores two distinct projection mechanisms: Q-Formers and Linear Projections (LPs). Through extensive experimentation across all four modalities on 16 benchmarks, we explore both methods and assess their adaptability in integrated and separate cross-modal reasoning. The Q-Former projection demonstrates superior performance in single modality scenarios and adaptability in joint versus discriminative reasoning involving two or more modalities. However, it exhibits lower generalization capabilities than linear projection in contexts where task-modality data are limited. To enable this framework, we devise a scalable pipeline that automatically generates high-quality, instruction-tuning datasets from readily available captioning data across different modalities, and contribute 24K QA data for audio and 250K QA data for 3D. To facilitate further research in cross-modal reasoning, we introduce the DisCRn (Discriminative Cross-modal Reasoning) benchmark comprising 9K audio-video QA samples and 28K image-3D QA samples that require the model to reason discriminatively across disparate input modalities.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 92 of its 129 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

92 of 129 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions20 Nov

Other references · 91

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2016Learning without Forgettingcited by 2,496
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  16. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  17. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  18. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  20. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  21. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  22. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  23. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  25. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  26. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  27. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  28. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  29. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  30. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  31. 2018nocaps: novel object captioning at scalecited by 444
  32. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  33. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  34. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  35. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  36. 2019Clotho: An Audio Captioning Datasetcited by 358
  37. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  38. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 34; 2026: 12.

80 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  4. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  5. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  7. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  8. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  9. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  10. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  11. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  12. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
38 more
  1. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  2. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  3. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  4. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  5. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  6. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  7. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  8. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  9. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  10. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  11. 2024The Evolution of Multimodal Model Architecturescited by 21
  12. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  13. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  14. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  15. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 18
  16. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  17. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understandingcited by 16
  18. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  19. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  20. 2024MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Drivingcited by 13
  21. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  22. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 13
  23. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 13
  24. 2024Model Composition for Multimodal Large Language Modelscited by 12
  25. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  26. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  27. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  28. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  29. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  30. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  31. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 8
  32. 2025Mitigating Modal Imbalance in Multimodal Reasoningcited by 8
  33. 2024LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Livingcited by 7
  34. 2024Affective Computing Has Changed: The Foundation Model Disruptioncited by 7
  35. 2024Skipping Computations in Multimodal LLMscited by 7
  36. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  37. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6
  38. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Modelscited by 6

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
  2. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
  3. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
  4. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMs
  5. 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
  6. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
  7. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoning
  8. 2026Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory