mlsanityData to 14 Oct 2026

Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models

arXiv 2412.09645 ↗First posted 10 Dec 2024cs.CV cs.AI cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

32 of 45 references.

References · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
20 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2017Video Enhancement with Task-Oriented Flowcited by 619
  9. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  10. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  12. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  13. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  14. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  15. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 159
  16. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  17. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  18. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  19. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  20. 2024Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agentcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 18.

41 papers.

Sort:

By area:

  1. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  2. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  3. 2025SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentioncited by 58
  4. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
  5. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 45
  6. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  7. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  8. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 31
  9. 2025VABench: A Comprehensive Benchmark for Audio-Video Generationcited by 30
  10. 2025Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmarkcited by 29
  11. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  12. 2026SLA2: Sparse-Linear Attention with Learnable Routing and QATcited by 22
29 more
  1. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 20
  2. 2026Agent-as-a-Judgecited by 17
  3. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  4. 2026VQQA: An Agentic Approach for Video Evaluation and Quality Improvementcited by 12
  5. 2025Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusabilitycited by 11
  6. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Modelscited by 9
  7. 2025EgoX: Egocentric Video Generation from a Single Exocentric Videocited by 8
  8. 2025GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoningcited by 7
  9. 2026VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generationcited by 7
  10. 2026MultiAnimate: Pose-Guided Image Animation Made Extensiblecited by 6
  11. 2026Benchmark Everything Everywhere All at Oncecited by 6
  12. 2025LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localizationcited by 5
  13. 2025CoCo4D: Comprehensive and Complex 4D Scene Generationcited by 4
  14. 2025AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluationcited by 4
  15. 2025Coupled Diffusion Sampling for Training-Free Multi-View Image Editingcited by 4
  16. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 4
  17. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 4
  18. 2026Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatarscited by 3
  19. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  20. 2026Predict to Skip: Linear Multistep Feature Forecasting for Efficient Diffusion Transformerscited by 2
  21. 2026TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generationcited by 2
  22. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  23. 2025When and Where do Events Switch in Multi-Event Video Generation?cited by 1
  24. 2025MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMscited by 0
  25. 2026Intelligent Co-Design: An Interactive LLM Framework for Interior Spatial Design via Multi-Modal Agentscited by 0
  26. 2026TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructionscited by 0
  27. 2026V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learningcited by 0
  28. 2026Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generationcited by 0
  29. 2026Parameterized Stripe Attention for Efficient Video Generationcited by 0