mlsanityData to 14 Oct 2026

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

arXiv 2308.09936 ↗First posted 19 Aug 2023cs.CV cs.AI cs.CLcited by 125

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. Our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 62 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

48 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 46

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
34 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  15. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  16. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  17. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  18. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  19. 2016Visual Dialogcited by 377
  20. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  21. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  22. 2022Visual Spatial Reasoningcited by 293
  23. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  24. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  25. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  26. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  27. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  28. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  29. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  30. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  31. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  32. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 61
  33. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54
  34. 2023Improving Cross-Task Generalization with Step-by-Step Instructionscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 60; 2025: 39; 2026: 12.

125 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  3. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  7. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  8. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  9. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  10. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  11. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
  12. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
38 more
  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  2. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  3. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  4. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 50
  5. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 46
  6. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 43
  7. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  8. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 33
  11. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  12. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  13. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  14. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  15. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  16. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  17. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  18. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 23
  19. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  20. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  21. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 22
  22. 2024The Evolution of Multimodal Model Architecturescited by 21
  23. 2025Benchmarking Visual Language Models on Standardized Visualization Literacy Testscited by 20
  24. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  25. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  26. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 19
  27. 2024Plug-and-Play Grounding of Reasoning in Multimodal Large Language Modelscited by 18
  28. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  29. 2024VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimizationcited by 16
  30. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Modelscited by 16
  31. 2024DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Modelscited by 15
  32. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Modelscited by 15
  33. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capabilitycited by 15
  34. 2023Assessing GPT4-V on Structured Reasoning Taskscited by 14
  35. 2024Exploring the Capabilities of Large Multimodal Models on Dense Textcited by 14
  36. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12
  37. 2024Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-bodycited by 12
  38. 2025Can Multimodal LLMs Perform Time Series Anomaly Detection?cited by 12

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts
  2. 2026ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering
  3. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
  4. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  5. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  6. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  7. 2025SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond