# Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models

arXiv: https://arxiv.org/abs/2412.09645 · HTML: https://mlsanity.com/paper/2412.09645/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](https://mlsanity.com/paper/2407.14505/) (2024)
- [SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning](https://mlsanity.com/paper/2308.00436/) (2023)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [Autonomous Evaluation and Refinement of Digital Agents](https://mlsanity.com/paper/2404.06474/) (2024)
- [MMInA: Benchmarking Multihop Multimodal Internet Agents](https://mlsanity.com/paper/2404.09992/) (2024)

## Built on by (most cited first, 30 of 41)

- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer](https://mlsanity.com/paper/2509.24695/) (2025)
- [SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention](https://mlsanity.com/paper/2509.24006/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation](https://mlsanity.com/paper/2511.04675/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [InstructX: Towards Unified Visual Editing with MLLM Guidance](https://mlsanity.com/paper/2510.08485/) (2025)
- [VISTA: A Test-Time Self-Improving Video Generation Agent](https://mlsanity.com/paper/2510.15831/) (2025)
- [VABench: A Comprehensive Benchmark for Audio-Video Generation](https://mlsanity.com/paper/2512.09299/) (2025)
- [Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark](https://mlsanity.com/paper/2510.13759/) (2025)
- [VideoGen-Eval: Agent-based System for Video Generation Evaluation](https://mlsanity.com/paper/2503.23452/) (2025)
- [SLA2: Sparse-Linear Attention with Learnable Routing and QAT](https://mlsanity.com/paper/2602.12675/) (2026)
- [Playing with Transformer at 30+ FPS via Next-Frame Diffusion](https://mlsanity.com/paper/2506.01380/) (2025)
- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [VQQA: An Agentic Approach for Video Evaluation and Quality Improvement](https://mlsanity.com/paper/2603.12310/) (2026)
- [Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability](https://mlsanity.com/paper/2512.05394/) (2025)
- [AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models](https://mlsanity.com/paper/2509.26100/) (2025)
- [EgoX: Egocentric Video Generation from a Single Exocentric Video](https://mlsanity.com/paper/2512.08269/) (2025)
- [GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning](https://mlsanity.com/paper/2503.02341/) (2025)
- [VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation](https://mlsanity.com/paper/2606.08091/) (2026)
- [MultiAnimate: Pose-Guided Image Animation Made Extensible](https://mlsanity.com/paper/2602.21581/) (2026)
- [Benchmark Everything Everywhere All at Once](https://mlsanity.com/paper/2606.06462/) (2026)
- [LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization](https://mlsanity.com/paper/2512.02933/) (2025)
- [CoCo4D: Comprehensive and Complex 4D Scene Generation](https://mlsanity.com/paper/2506.19798/) (2025)
- [AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation](https://mlsanity.com/paper/2507.01255/) (2025)
- [Coupled Diffusion Sampling for Training-Free Multi-View Image Editing](https://mlsanity.com/paper/2510.14981/) (2025)
- [MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation](https://mlsanity.com/paper/2605.28035/) (2026)
- [HarnessEval-W: Agentifying the Evaluation of Visual Worlds](https://mlsanity.com/paper/2608.16859/) (2026)
- [Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars](https://mlsanity.com/paper/2602.01538/) (2026)

## Abstract (arXiv)

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.