mlsanityData to 14 Oct 2026

DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks

arXiv 2309.17167 ↗First posted 29 Sep 2023cs.AI cs.CL cs.LGcited by 101

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance in various evaluation benchmarks. However, concerns are raised about potential data contamination in their considerable volume of training corpus. Moreover, the static nature and fixed complexity of current benchmarks may inadequately gauge the advancing capabilities of LLMs. In this paper, we introduce DyVal, a general and flexible protocol for dynamic evaluation of LLMs. Based on our framework, we build graph-informed DyVal by leveraging the structural advantage of directed acyclic graphs to dynamically generate evaluation samples with controllable complexities. DyVal generates challenging evaluation sets on reasoning tasks including mathematics, logical reasoning, and algorithm problems. We evaluate various LLMs ranging from Flan-T5-large to GPT-3.5-Turbo and GPT-4. Experiments show that LLMs perform worse in DyVal-generated evaluation samples with different complexities, highlighting the significance of dynamic evaluation. We also analyze the failure cases and results of different prompting methods. Moreover, DyVal-generated samples are not only evaluation sets, but also helpful data for fine-tuning to improve the performance of LLMs on existing benchmarks. We hope that DyVal can shed light on future evaluation research of LLMs. Code is available at: https://github.com/microsoft/promptbench.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 58 of its 84 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

58 of 84 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Cumulative Reasoning with Large Language Models8 Aug
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  3. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
38 more
  1. 2017A Closer Look at Memorization in Deep Networkscited by 942
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  9. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  13. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  14. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  15. 2016Solving General Arithmetic Word Problemscited by 360
  16. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  17. 2020Transformers as Soft Reasoners over Languagecited by 270
  18. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  19. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  20. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  21. 2022Data Contamination: From Memorization to Exploitationcited by 236
  22. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  23. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  24. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  25. 2019Good-Enough Compositional Data Augmentationcited by 158
  26. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 150
  27. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  28. 2023Can Large Language Models Infer Causation from Correlation?cited by 146
  29. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  30. 2014Grammar as a Foreign Languagecited by 70
  31. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 63
  32. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  33. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  34. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  35. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  36. 2022Adaptive Testing of Computer Vision Modelscited by 39
  37. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  38. 2023Pretraining on the Test Set Is All You Needcited by 33

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 47; 2026: 31.

101 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  3. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  4. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  5. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  6. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  7. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  8. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  9. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  10. 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Modelscited by 35
  11. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  12. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 27
38 more
  1. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  2. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  3. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 23
  4. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 21
  5. 2024Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Miningcited by 20
  6. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 16
  7. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 15
  8. 2024TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planningcited by 14
  9. 2024Co-occurrence is not Factual Association in Language Modelscited by 12
  10. 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarkingcited by 12
  11. 2024DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoningcited by 11
  12. 2025Automated Capability Discovery via Foundation Model Self-Explorationcited by 10
  13. 2024BenchAgents: Multi-Agent Systems for Structured Benchmark Creationcited by 9
  14. 2025InductionBench: LLMs Fail in the Simplest Complexity Classcited by 9
  15. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 9
  16. 2026ProbeLLM: Automating Principled Diagnosis of LLM Failurescited by 9
  17. 2024ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Modelscited by 8
  18. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  19. 2025LLM-Powered Benchmark Factory: Reliable, Generic, and Efficientcited by 7
  20. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  21. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  22. 2024Reliable and diverse evaluation of LLM medical knowledge masterycited by 6
  23. 2025Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Basecited by 6
  24. 2025Generative Evaluation of Complex Reasoning in Large Language Modelscited by 6
  25. 2025Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluationcited by 6
  26. 2025How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation under the One-Time-Pad-Based Frameworkcited by 6
  27. 2025Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verificationcited by 6
  28. 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systemscited by 6
  29. 2026Benchmark Everything Everywhere All at Oncecited by 6
  30. 2025Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasetscited by 5
  31. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  32. 2026WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmarkcited by 5
  33. 2024TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domainscited by 4
  34. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspectivecited by 4
  35. 2025Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacementscited by 4
  36. 2025Adaptively profiling models with task elicitationcited by 4
  37. 2025Framing the Game: How Context Shapes LLM Decision-Makingcited by 4
  38. 2025AcademicEval: Live Long-Context LLM Benchmarkcited by 4

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  3. 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning
  4. 2024Are Your LLMs Capable of Stable Reasoning?
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  7. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
  8. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models