mlsanityData to 7 Oct 2026

ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models

arXiv 2403.20194 ↗First posted 29 Mar 2024cs.MMcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper presents ConvBench, a novel multi-turn conversation evaluation benchmark tailored for Large Vision-Language Models (LVLMs). Unlike existing benchmarks that assess individual capabilities in single-turn dialogues, ConvBench adopts a three-level multimodal capability hierarchy, mimicking human cognitive processes by stacking up perception, reasoning, and creativity. Each level focuses on a distinct capability, mirroring the cognitive progression from basic perception to logical reasoning and ultimately to advanced creativity. ConvBench comprises 577 meticulously curated multi-turn conversations encompassing 215 tasks reflective of real-world demands. Automatic evaluations quantify response performance at each turn and overall conversation level. Leveraging the capability hierarchy, ConvBench enables precise attribution of conversation mistakes to specific levels. Experimental results reveal a performance gap between multi-modal models, including GPT4-V, and human performance in multi-turn conversations. Additionally, weak fine-grained perception in multi-modal models contributes to reasoning and creation failures. ConvBench serves as a catalyst for further research aimed at enhancing visual dialogues.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 53 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

27 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  7. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
1 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 14

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
2 more
  1. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  2. 2023IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Modelscited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2025Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?cited by 22
  7. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  8. 2025MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversationscited by 12
  9. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  10. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  11. 2025SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoningcited by 3
  12. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
12 more
  1. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Modelscited by 3
  2. 2025REEF: Relevance-Aware and Efficient LLM Adapter for Video Understandingcited by 2
  3. 2025AgriDoctor: A Multimodal Intelligent Assistant for Agriculturecited by 2
  4. 2026Rethinking Token Reduction for Large Vision-Language Modelscited by 2
  5. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  6. 2026First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Modelscited by 1
  7. 2024Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimationcited by 0
  8. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0
  9. 2026Advancing Creative Physical Intelligence in Large Multimodal Modelscited by 0
  10. 2026Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactionscited by 0
  11. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  12. 2026SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models
  2. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  3. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models
  4. 2025MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning
  5. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
  6. 2025MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation
  7. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following
  8. 2025ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models