mlsanityData to 7 Oct 2026

Dynamic Evaluation of Large Language Models by Meta Probing Agents

arXiv 2402.14865 ↗First posted 21 Feb 2024cs.CL cs.AI cs.LGcited by 54

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Evaluation of large language models (LLMs) has raised great concerns in the community due to the issue of data contamination. Existing work designed evaluation protocols using well-defined algorithms for specific tasks, which cannot be easily extended to diverse scenarios. Moreover, current evaluation benchmarks can only provide the overall benchmark results and cannot support a fine-grained and multifaceted analysis of LLMs' abilities. In this paper, we propose meta probing agents (MPA), a general dynamic evaluation protocol inspired by psychometrics to evaluate LLMs. MPA is the key component of DyVal 2, which naturally extends the previous DyVal~citepzhu2023dyval. MPA designs the probing and judging agents to automatically transform an original evaluation problem into a new one following psychometric theory on three basic cognitive abilities: language understanding, problem solving, and domain knowledge. These basic abilities are also dynamically configurable, allowing multifaceted analysis. We conducted extensive evaluations using MPA and found that most LLMs achieve poorer performance, indicating room for improvement. Our multifaceted analysis demonstrated the strong correlation between the basic abilities and an implicit Matthew effect on model size, i.e., larger models possess stronger correlations of the abilities. MPA can also be used as a data augmentation approach to enhance LLMs. Code is available at: https://github.com/microsoft/promptbench.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 44 of its 60 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

44 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
31 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  7. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  10. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  12. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  13. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  14. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  15. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  16. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  17. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  18. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  19. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  20. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  21. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  22. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  23. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  24. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  25. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  26. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  27. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 46
  28. 2022Adaptive Testing of Computer Vision Modelscited by 39
  29. 2023Revealing the structure of language model capabilitiescited by 35
  30. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutationscited by 24
  31. 2023S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 30; 2026: 8.

54 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  3. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  4. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  5. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  6. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  7. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  8. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  9. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  10. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  11. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  12. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
38 more
  1. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 22
  2. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  3. 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoningcited by 17
  4. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  5. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
  6. 2025Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Searchcited by 10
  7. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Modelscited by 8
  8. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  9. 2025DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generationcited by 7
  10. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Modelscited by 7
  11. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 7
  12. 2024Reliable and diverse evaluation of LLM medical knowledge masterycited by 6
  13. 2025LLM-Powered Benchmark Factory: Reliable, Generic, and Efficientcited by 6
  14. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  15. 2025Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluationcited by 6
  16. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  17. 2025Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verificationcited by 6
  18. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  19. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
  20. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  21. 2026WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmarkcited by 4
  22. 2025Unbiased Evaluation of Large Language Models from a Causal Perspectivecited by 3
  23. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  24. 2025SDEval: Safety Dynamic Evaluation for Multimodal Large Language Modelscited by 3
  25. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructionscited by 3
  26. 2026Robust Reasoning Benchmarkcited by 3
  27. 2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
  28. 2025Large Language Models Could Be Rote Learnerscited by 2
  29. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  30. 2025Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Eracited by 2
  31. 2025MACEval: A Multi-Agent Continual Evaluation Network for Large Modelscited by 2
  32. 2026Enhancing Research Idea Generation through Combinatorial Innovation and Multi-Agent Iterative Search Strategiescited by 2
  33. 2024Project MPG: towards a generalized performance benchmark for LLM capabilitiescited by 1
  34. 2024Addressing Data Leakage in HumanEval Using Combinatorial Test Designcited by 1
  35. 2025AdEval: Alignment-based Dynamic Evaluation to Mitigate Data Contamination in Large Language Modelscited by 1
  36. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  37. 2026From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoningcited by 1
  38. 2025LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluationcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  2. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
  3. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  5. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  6. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  7. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  8. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research