mlsanityData to 14 Oct 2026

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

arXiv 2308.06595 ↗First posted 12 Aug 2023cs.CL cs.AI cs.CVcited by 102

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 100 references that are in the map. First posted on arXiv 12 Aug 2023.

Builds on

79 of 100 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  6. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment20 Dec

Other references · 73

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  14. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  16. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  17. 2021Grounded Language-Image Pre-trainingcited by 1,156
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  19. 2023LIMA: Less Is More for Alignmentcited by 1,100
  20. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  21. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  22. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  23. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  24. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  25. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  26. 2023Instruction Tuning with GPT-4cited by 686
  27. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  28. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  29. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  30. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  31. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  32. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  33. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  34. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  35. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  36. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  37. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  38. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 47; 2025: 27; 2026: 13.

102 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  4. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  6. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  7. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  8. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  9. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  10. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  11. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  12. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
38 more
  1. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  2. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  4. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  5. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  7. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  8. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  9. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  12. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  13. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  14. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 55
  15. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 51
  16. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 44
  17. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  18. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  19. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  20. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  21. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36
  22. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  23. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  24. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  25. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  26. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chaincited by 29
  27. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  28. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  29. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  30. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  31. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Modelscited by 25
  32. 2024Leveraging Vision Language Models for Specialized Agricultural Taskscited by 25
  33. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  34. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  35. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  36. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  37. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  38. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiadscited by 16

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
  2. 2025Vision Language Models: A Survey of 26K Papers
  3. 2026MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
  4. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  5. 2026LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Models
  6. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  7. 2025Instruction-Following Evaluation of Large Vision-Language Models
  8. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models