mlsanityData to 14 Oct 2026

VisionArena: 230K Real World User-VLM Conversations with Preference Labels

arXiv 2412.08687 ↗First posted 11 Dec 2024cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With the growing adoption and capabilities of vision-language models (VLMs) comes the need for benchmarks that capture authentic user-VLM interactions. In response, we create VisionArena, a dataset of 230K real-world conversations between users and VLMs. Collected from Chatbot Arena - an open-source platform where users interact with VLMs and submit preference votes - VisionArena spans 73K unique users, 45 VLMs, and 138 languages. Our dataset contains three subsets: VisionArena-Chat, 200k single and multi-turn conversations between a user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark of 500 diverse user prompts that efficiently approximate the live Chatbot Arena model rankings. Additionally, we highlight the types of question asked by users, the influence of response style on preference, and areas where models often fail. We find open-ended tasks like captioning and humor are highly style-dependent, and current VLMs struggle with spatial reasoning and planning tasks. Lastly, we show finetuning the same base model on VisionArena-Chat outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 54 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

31 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences16 Jun
  2. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation12 Jan
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
16 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  7. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  10. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  11. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  12. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  13. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  14. 2024WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wildcited by 20
  15. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  16. 2024VibeCheck: Discover and Quantify Qualitative Differences in Large Language Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 13.

22 papers.

Sort:

By area:

  1. 2025MiMo-VL Technical Reportcited by 115
  2. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  3. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
  4. 2026STEP3-VL-10B Technical Reportcited by 32
  5. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 23
  6. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 20
  7. 2026KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companionscited by 11
  8. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  9. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 8
  10. 2025LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Modelscited by 5
  11. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  12. 2026K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judgecited by 4
10 more
  1. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 2
  2. 2026SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Modelscited by 2
  3. 2026A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulationcited by 2
  4. 2026PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perceptioncited by 1
  5. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessmentcited by 1
  6. 2025Feedback Forensics: A Toolkit to Measure AI Personalitycited by 0
  7. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  8. 2026Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modelingcited by 0
  9. 2026Towards High-Level Semantic Intelligencecited by 0
  10. 2026A Statistical Audit of Physical AI Benchmark Redundancycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Dataset for Dynamic Human Preferences for Vision Language Models
  2. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  3. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
  4. 2025V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction
  5. 2024VHELM: A Holistic Evaluation of Vision Language Models
  6. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  7. 2024What matters when building vision-language models?
  8. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model