mlsanityData to 7 Oct 2026

Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation

arXiv 2402.11443 ↗First posted 18 Feb 2024cs.CLcited by 73

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

This paper presents a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models (LLMs), aiming for a more accurate assessment of their capabilities and limitations. We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence that dynamically extend existing benchmarks. Towards a more scalable, robust and fine-grained evaluation, we implement six reframing operations to construct evolving instances testing LLMs against diverse queries, data noise and probing their problem-solving sub-abilities. With this framework, we extend benchmark datasets of four tasks. Experimental results show a general performance decline in most LLMs against their original results. This decline under our scalable and robust evaluations, alongside our fine-grained evaluation, more accurately reflect models' capabilities. Besides, our framework widens performance discrepancies both between different models and within the same model across various tasks, facilitating more informed model selection for specific tasks (Code and data are available at https://github.com/NanshineLoong/Self-Evolving-Benchmark).

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 25 of its 46 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

25 of 46 references.

References · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
13 more
  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  3. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  4. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  5. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  6. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  7. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  8. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  9. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  10. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  11. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Promptscited by 34
  12. 2023Making Large Language Models Better Data Creatorscited by 28
  13. 2023Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Taskcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 37; 2026: 17.

73 papers.

Sort:

By area:

  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  4. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  5. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  6. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  7. 2025UFO2: The Desktop AgentOScited by 38
  8. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  9. 2024A Survey of Useful LLM Evaluationcited by 28
  10. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  11. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  12. 2024How secure is AI-generated Code: A Large-Scale Comparison of Large Language Modelscited by 23
38 more
  1. 2025EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Frameworkcited by 23
  2. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  3. 2025Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarkingcited by 21
  4. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  5. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  6. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  7. 2025CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answeringcited by 15
  8. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  9. 2024LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluationcited by 12
  10. 2024DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoningcited by 11
  11. 2026MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systemscited by 11
  12. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 10
  13. 2024AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Modelscited by 9
  14. 2024A Multi-LLM Debiasing Frameworkcited by 9
  15. 2025Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generationcited by 9
  16. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  17. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Modelscited by 8
  18. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 8
  19. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Modelscited by 7
  20. 2025DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generationcited by 7
  21. 2025BetaWeb: Towards a Blockchain-enabled Trustworthy Agentic Webcited by 7
  22. 2024BenchAgents: Multi-Agent Systems for Structured Benchmark Creationcited by 6
  23. 2025Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Basecited by 6
  24. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  25. 2026AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignmentcited by 6
  26. 2024Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidencecited by 5
  27. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Makingcited by 5
  28. 2025Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigmcited by 5
  29. 2026EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environmentscited by 5
  30. 2025Adaptively profiling models with task elicitationcited by 4
  31. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  32. 2025DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Responsecited by 4
  33. 2026PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Reviewcited by 4
  34. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 3
  35. 2025Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Taskscited by 3
  36. 2026Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interactioncited by 3
  37. 2026LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?cited by 3
  38. 2024Balancing Rigor and Utility: Mitigating Cognitive Biases in Large Language Models for Multiple-Choice Questionscited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026When Is Enough Enough in Self-Evolving LLM Systems?
  2. 2024A Survey on Self-Evolution of Large Language Models
  3. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?
  4. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle
  5. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  6. 2023AgentBench: Evaluating LLMs as Agents
  7. 2026PACE: Two-Timescale Self-Evolution for Small Language Model Agents
  8. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities