# VisionArena: 230K Real World User-VLM Conversations with Preference Labels

arXiv: https://arxiv.org/abs/2412.08687 · HTML: https://mlsanity.com/paper/2412.08687/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [Yin and Yang: Balancing and Answering Binary Visual Questions](https://mlsanity.com/paper/1511.05099/) (2015)
- [WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences](https://mlsanity.com/paper/2406.11069/) (2024)
- [Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models](https://mlsanity.com/paper/2404.12387/) (2024)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild](https://mlsanity.com/paper/2409.03753/) (2024)
- [LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content](https://mlsanity.com/paper/2410.10783/) (2024)

## Built on by (most cited first, 22 of 22)

- [MiMo-VL Technical Report](https://mlsanity.com/paper/2506.03569/) (2025)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era](https://mlsanity.com/paper/2503.12329/) (2025)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios](https://mlsanity.com/paper/2602.23166/) (2026)
- [SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks](https://mlsanity.com/paper/2507.01001/) (2025)
- [KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions](https://mlsanity.com/paper/2601.04745/) (2026)
- [HueManity: Probing Fine-Grained Visual Perception in MLLMs](https://mlsanity.com/paper/2506.03194/) (2025)
- [Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings](https://mlsanity.com/paper/2508.11847/) (2025)
- [LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models](https://mlsanity.com/paper/2502.14834/) (2025)
- [Visual Preference Optimization with Rubric Rewards](https://mlsanity.com/paper/2604.13029/) (2026)
- [K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge](https://mlsanity.com/paper/2602.09411/) (2026)
- [Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training](https://mlsanity.com/paper/2505.08971/) (2025)
- [SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models](https://mlsanity.com/paper/2604.17041/) (2026)
- [A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation](https://mlsanity.com/paper/2605.15761/) (2026)
- [PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception](https://mlsanity.com/paper/2606.28322/) (2026)
- [Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment](https://mlsanity.com/paper/2608.29696/) (2026)
- [Feedback Forensics: A Toolkit to Measure AI Personality](https://mlsanity.com/paper/2509.26305/) (2025)
- [Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Models](https://mlsanity.com/paper/2603.27522/) (2026)
- [Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling](https://mlsanity.com/paper/2604.05445/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [A Statistical Audit of Physical AI Benchmark Redundancy](https://mlsanity.com/paper/2608.25940/) (2026)

## Abstract (arXiv)

With the growing adoption and capabilities of vision-language models (VLMs) comes the need for benchmarks that capture authentic user-VLM interactions. In response, we create VisionArena, a dataset of 230K real-world conversations between users and VLMs. Collected from Chatbot Arena - an open-source platform where users interact with VLMs and submit preference votes - VisionArena spans 73K unique users, 45 VLMs, and 138 languages. Our dataset contains three subsets: VisionArena-Chat, 200k single and multi-turn conversations between a user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark of 500 diverse user prompts that efficiently approximate the live Chatbot Arena model rankings. Additionally, we highlight the types of question asked by users, the influence of response style on preference, and areas where models often fail. We find open-ended tasks like captioning and humor are highly style-dependent, and current VLMs struggle with spatial reasoning and planning tasks. Lastly, we show finetuning the same base model on VisionArena-Chat outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai