mlsanityData to 7 Oct 2026

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models

arXiv 2405.02287 ↗First posted 3 May 2024cs.CL cs.AI cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models. Vibe-Eval consists of 269 visual understanding prompts, including 100 of hard difficulty, complete with gold-standard responses authored by experts. Vibe-Eval is open-ended and challenging with dual objectives: (i) vibe checking multimodal chat models for day-to-day tasks and (ii) rigorously testing and probing the capabilities of present frontier models. Notably, our hard set contains >50% questions that all frontier models answer incorrectly. We explore the nuances of designing, evaluating, and ranking models on ultra challenging prompts. We also discuss trade-offs between human and automatic evaluation, and show that automatic model evaluation using Reka Core roughly correlates to human judgment. We offer free API access for the purpose of lightweight evaluation and plan to conduct formal human evaluations for public models that perform well on the Vibe-Eval's automatic scores. We release the evaluation code and data, see https://github.com/reka-ai/reka-vibe-eval

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 25 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

14 of 25 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  11. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  12. 2021The Benchmark Lotterycited by 133
2 more
  1. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  2. 2023Pretraining on the Test Set Is All You Needcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 8; 2026: 4.

35 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  3. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  4. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  5. 2025MiMo-VL Technical Reportcited by 106
  6. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  7. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  8. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  11. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  12. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
23 more
  1. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  2. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videoscited by 51
  3. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  4. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  5. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 18
  6. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  7. 2025CountQA: How Well Do MLLMs Count in the Wild?cited by 17
  8. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  9. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  10. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  11. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  12. 2024Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Modelscited by 9
  13. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  14. 2024NL-Eye: Abductive NLI for Imagescited by 5
  15. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  16. 2024Program Synthesis Benchmark for Visual Programming in XLogoOnline Environmentcited by 4
  17. 2024What Are They Doing? Joint Audio-Speech Co-Reasoningcited by 4
  18. 2024GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Modelscited by 3
  19. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3
  20. 2025HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoningcited by 0
  21. 2026Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMscited by 0
  22. 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Modelscited by 0
  23. 2026Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
  2. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  3. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
  4. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  5. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  7. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models
  8. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models