MACEval: A Multi-Agent Continual Evaluation Network for Large Models
Not on the map. Why?
Abstract (arXiv)
Hundreds of benchmarks dedicated to evaluating large models have been presented over the past few years. However, most of them remain closed-ended and are prone to overfitting due to the potential data contamination. Moreover, the increasing scale and scope of current benchmarks with transient metrics, as well as the heavily human-dependent curation procedure, pose significant challenges for timely maintenance and adaptation. In this paper, we introduce MACEval, a Multi-Agent Continual Evaluation network for dynamic evaluation of large models, and define new metrics to quantify performance longitudinally. MACEval employs an interactive and autonomous evaluation mode, utilizing role assignment, in-process data generation, and evaluation routing through a cascaded agent network. Extensive experiments on 23 large models demonstrate the effectiveness of MACEval, which also lightens the evaluation process and reduces a considerable amount of overhead. We hope that MACEval can broaden future directions of large model evaluation. Project page: https://github.com/zijianchen98/MACEval.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 55 of its 71 references that are in the map. First posted on arXiv 12 Nov 2025.
Builds on
55 of 71 references.
References · 55
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2024Qwen2 Technical Reportcited by 2,120
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
- 2023A Survey on Evaluation of Large Language Modelscited by 1,242
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023Teaching Large Language Models to Self-Debugcited by 988
- 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
- 2022Quantifying Memorization Across Neural Language Modelscited by 763
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
- 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
- 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
- 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
- 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
- 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
- 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 162
- 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
- 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
- 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
- 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
- 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
- 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
- 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 75
- 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
- 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
- 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
- 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
- 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
- 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMscited by 33
- 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
- 2024Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalizationcited by 24
- 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.