mlsanityData to 7 Oct 2026

TouchStone: Evaluating Vision-Language Models by Language Models

arXiv 2308.16890 ↗First posted 31 Aug 2023cs.CV cs.CLcited by 54

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models (LVLMs) have recently witnessed rapid advancements, exhibiting a remarkable capacity for perceiving, understanding, and processing visual information by connecting visual receptor with large language models (LLMs). However, current assessments mainly focus on recognizing and reasoning abilities, lacking direct evaluation of conversational skills and neglecting visual storytelling abilities. In this paper, we propose an evaluation method that uses strong LLMs as judges to comprehensively evaluate the various abilities of LVLMs. Firstly, we construct a comprehensive visual dialogue dataset TouchStone, consisting of open-world images and questions, covering five major categories of abilities and 27 subtasks. This dataset not only covers fundamental recognition and comprehension but also extends to literary creation. Secondly, by integrating detailed image annotations we effectively transform the multimodal input content into a form understandable by LLMs. This enables us to employ advanced LLMs for directly evaluating the quality of the multimodal dialogue without requiring human intervention. Through validation, we demonstrate that powerful LVLMs, such as GPT-4, can effectively score dialogue quality by leveraging their textual capabilities alone, aligning with human preferences. We hope our work can serve as a touchstone for LVLMs' evaluation and pave the way for building stronger LVLMs. The evaluation code is available at https://github.com/OFA-Sys/TouchStone.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

38 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
20 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  19. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  20. 2022OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 28; 2025: 17; 2026: 2.

54 papers.

Sort:

By area:

  1. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  4. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  5. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  6. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  7. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  8. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  9. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  10. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  11. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  12. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
38 more
  1. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  2. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  3. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  4. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  5. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  8. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  9. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  10. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  11. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Modelscited by 30
  12. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  13. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  14. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchycited by 26
  15. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  16. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  17. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 17
  18. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 15
  19. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14
  20. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  21. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  22. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Modelscited by 10
  23. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  24. 2025ArtMentor: AI-Assisted Evaluation of Artworks to Explore Multimodal Large Language Models Capabilitiescited by 9
  25. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  26. 2024Towards Artwork Explanation in Large-scale Vision Language Modelscited by 8
  27. 2024AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Modelscited by 8
  28. 2024MULTI: Multimodal Understanding Leaderboard with Text and Imagescited by 7
  29. 2024IDEA-Bench: How Far are Generative Models from Professional Designing?cited by 7
  30. 2024How Culturally Aware are Vision-Language Models?cited by 6
  31. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  32. 2024LightHouse: A Survey of AGI Hallucinationcited by 4
  33. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  34. 2024Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytellingcited by 3
  35. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  36. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  37. 2025MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosiscited by 2
  38. 2024Can Large Language Models Automatically Jailbreak GPT-4V?cited by 1

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  2. 2024Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models
  3. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  4. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  5. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  8. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models