mlsanityData to 7 Oct 2026

GenAI Arena: An Open Evaluation Platform for Generative Models

arXiv 2406.04485 ↗First posted 6 Jun 2024cs.AI cs.CVcited by 81

Not on the map. Why?

Cited by 27 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative AI has made remarkable strides to revolutionize fields such as image and video generation. These advancements are driven by innovative algorithms, architecture, and data. However, the rapid proliferation of generative models has highlighted a critical gap: the absence of trustworthy evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc often fail to capture the nuanced quality and user satisfaction associated with generative outputs. This paper proposes an open platform GenAI-Arena to evaluate different image and video generative models, where users can actively participate in evaluating these models. By leveraging collective user feedback and votes, GenAI-Arena aims to provide a more democratic and accurate measure of model performance. It covers three tasks of text-to-image generation, text-to-video generation, and image editing respectively. Currently, we cover a total of 35 open-source generative models. GenAI-Arena has been operating for seven months, amassing over 9000 votes from the community. We describe our platform, analyze the data, and explain the statistical methods for ranking the models. To further promote the research in building model-based evaluation metrics, we release a cleaned version of our preference data for the three tasks, namely GenAI-Bench. We prompt the existing multi-modal models like Gemini, and GPT-4o to mimic human voting. We compute the accuracy by comparing the model voting with the human voting to understand their judging abilities. Our results show existing multimodal models are still lagging in assessing the generated visual content, even the best model GPT-4o only achieves an average accuracy of 49.19 across the three generative tasks. Open-source MLLMs perform even worse due to the lack of instruction-following and reasoning ability in complex vision scenarios.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

57 of 74 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  13. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  14. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  17. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  18. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  19. 2023Adversarial Diffusion Distillationcited by 741
  20. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  21. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  22. 2023ModelScope Text-to-Video Technical Reportcited by 651
  23. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  24. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  25. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  26. 2023Zero-shot Image-to-Image Translationcited by 503
  27. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  28. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  29. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  30. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  31. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  32. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  33. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  34. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  35. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  36. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  37. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  38. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 36; 2026: 37.

81 papers.

Sort:

By area:

  1. 2025Improving Video Generation with Human Feedbackcited by 267
  2. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  3. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  4. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  5. 2024OmniEdit: Building Image Editing Generalist Models Through Specialist Supervisioncited by 145
  6. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  7. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  8. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 61
  9. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editingcited by 59
  10. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  11. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  12. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
38 more
  1. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  2. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 17
  3. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  4. 2025Gaussian Mixture Flow Matching Modelscited by 16
  5. 2025REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformationscited by 15
  6. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  7. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 14
  8. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 11
  9. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  10. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  11. 2026RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Timecited by 11
  12. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  13. 20253D Arena: An Open Platform for Generative 3D Evaluationcited by 8
  14. 2025Music Arena: Live Evaluation for Text-to-Musiccited by 8
  15. 2026Generative World Renderercited by 8
  16. 2025Visual Persona: Foundation Model for Full-Body Human Customizationcited by 7
  17. 2024MixEval-X: Any-to-Any Evaluations from Real-World Data Mixturescited by 6
  18. 2025The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Modelscited by 6
  19. 2024EvalGIM: A Library for Evaluating Generative Image Modelscited by 5
  20. 2025Understanding Generative AI Capabilities in Everyday Image Editing Taskscited by 5
  21. 2026Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modelingcited by 5
  22. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instructioncited by 5
  23. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  24. 2026RewardHarness: Self-Evolving Agentic Post-Trainingcited by 5
  25. 2025ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluationcited by 4
  26. 2025Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matricescited by 4
  27. 2026GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?cited by 4
  28. 2026EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editingcited by 4
  29. 2025What are Foundation Models Cooking in the Post-Soviet World?cited by 3
  30. 2025Multimodal Language Models as Text-to-Image Model Evaluatorscited by 3
  31. 2026K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judgecited by 3
  32. 2026Diffusion Templates: A Unified Plugin Framework for Controllable Diffusioncited by 3
  33. 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentratingcited by 3
  34. 2025DynamicEval: Rethinking Evaluation for Dynamic Text-to-Video Synthesiscited by 2
  35. 2025Envision: Benchmarking Unified Understanding & Generation for Causal World Process Insightscited by 2
  36. 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Modelcited by 2
  37. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generationcited by 2
  38. 2024Interactive Visual Assessment for Text-to-Image Generation Modelscited by 1

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
  2. 2024Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation
  3. 2024Benchmarking Generative AI Models for Deep Learning Test Input Generation
  4. 2025Generative AI for Film Creation: A Survey of Recent Advances
  5. 2024Generative AI for Visualization: State of the Art and Future Directions
  6. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation
  7. 2025Generative AI for Vision: A Comprehensive Study of Frameworks and Applications
  8. 2025AI-GenBench: A New Ongoing Benchmark for AI-Generated Image Detection