mlsanityData to 7 Oct 2026

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

arXiv 2406.11069 ↗First posted 16 Jun 2024cs.CV cs.AI cs.CLcited by 76

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent breakthroughs in vision-language models (VLMs) emphasize the necessity of benchmarking human preferences in real-world multimodal interactions. To address this gap, we launched WildVision-Arena (WV-Arena), an online platform that collects human preferences to evaluate VLMs. We curated WV-Bench by selecting 500 high-quality samples from 8,000 user submissions in WV-Arena. WV-Bench uses GPT-4 as the judge to compare each VLM with Claude-3-Sonnet, achieving a Spearman correlation of 0.94 with the WV-Arena Elo. This significantly outperforms other benchmarks like MMVet, MMMU, and MMStar. Our comprehensive analysis of 20K real-world interactions reveals important insights into the failure cases of top-performing VLMs. For example, we find that although GPT-4V surpasses many other models like Reka-Flash, Opus, and Yi-VL-Plus in simple visual recognition and reasoning tasks, it still faces challenges with subtle contextual cues, spatial reasoning, visual imagination, and expert domain knowledge. Additionally, current VLMs exhibit issues with hallucinations and safety when intentionally provoked. We are releasing our chat and feedback data to further advance research in the field of VLMs.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 83 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

48 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
33 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  14. 2016Modeling Context in Referring Expressionscited by 1,054
  15. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  16. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  17. 2016A Diagram Is Worth A Dozen Imagescited by 793
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  19. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  20. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  22. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  23. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  24. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  25. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  26. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  27. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  28. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  29. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  30. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  31. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  32. 2023Adapting Fake News Detection to the Era of Large Language Modelscited by 14
  33. 2024Improving Language Understanding from Screenshotscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 41; 2026: 19.

76 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  5. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  6. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  7. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  8. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  9. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  10. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  11. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  12. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
38 more
  1. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  2. 2025FineVision: Open Data Is All You Needcited by 51
  3. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  4. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  5. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 27
  6. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 26
  7. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 23
  8. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  9. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 22
  10. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  11. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 19
  12. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  13. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  14. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  15. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 17
  16. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  17. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  18. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  19. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  20. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13
  21. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
  22. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  23. 2024How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?cited by 8
  24. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  25. 2025Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoningcited by 8
  26. 2025Unified Reinforcement and Imitation Learning for Vision-Language Modelscited by 8
  27. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  28. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  29. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  30. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generationcited by 6
  31. 2025Can Large Models Fool the Eye? A New Turing Test for Biological Animationcited by 6
  32. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  33. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  34. 2025Visual Instruction Bottleneck Tuningcited by 5
  35. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  36. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  37. 2025Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editingcited by 4
  38. 2025PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Trickscited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VHELM: A Holistic Evaluation of Vision Language Models
  2. 2024What matters when building vision-language models?
  3. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  4. 2025CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
  5. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction
  6. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  7. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  8. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?