mlsanityData to 7 Oct 2026

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

arXiv 2306.17107 ↗First posted 29 Jun 2023cs.CV cs.CLcited by 272

Not on the map. Why?

Cited by 175 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

Key facts. Cited by 272 papers in the map (data through 7 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

41 of 67 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
29 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  14. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  15. 2018Object Hallucination in Image Captioningcited by 661
  16. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  17. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  18. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  19. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  20. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  21. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  22. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  23. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  24. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  25. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54
  26. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  27. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  28. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  29. 2023Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 137; 2025: 76; 2026: 21.

272 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
38 more
  1. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  2. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  8. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  9. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  10. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  11. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  12. 2024An Introduction to Vision-Language Modelingcited by 175
  13. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  14. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  15. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  16. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  17. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  18. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  19. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  20. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  21. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  22. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  23. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  24. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  25. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  26. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  27. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  28. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  29. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  30. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  31. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  32. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  33. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  34. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  35. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  36. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  37. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  38. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94

50 most cited of 272 All 272 (JSON)