mlsanityData to 7 Oct 2026

DEVAL: A Framework for Evaluating and Improving the Derivation Capability of Large Language Models

arXiv 2511.14813 ↗First posted 18 Nov 2025cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Assessing the reasoning ability of Large Language Models (LLMs) over data remains an open and pressing research question. Compared with LLMs, human reasoning can derive corresponding modifications to the output based on certain kinds of changes to the input. This reasoning pattern, which relies on abstract rules that govern relationships between changes of data, has not been comprehensively described or evaluated in LLMs. In this paper, we formally define this reasoning pattern as the Derivation Relation (DR) and introduce the concept of Derivation Capability (DC), i.e. applying DR by making the corresponding modification to the output whenever the input takes certain changes. To assess DC, a systematically constructed evaluation framework named DEVAL is proposed and used to evaluate five popular LLMs and one Large Reasoning Model in seven mainstream tasks. The evaluation results show that mainstream LLMs, such as GPT-4o and Claude3.5, exhibit moderate DR recognition capabilities but reveal significant drop-offs on applying DR effectively in problem-solving scenarios. To improve this, we propose a novel prompt engineering approach called Derivation Prompting (DP). It achieves an average improvement of 15.2% in DC for all tested LLMs, outperforming commonly used prompt engineering techniques.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 29 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

24 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability14 Feb
  2. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models14 Feb
  3. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  4. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May
  5. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks1 Mar
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct

Other references · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  5. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  6. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  7. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  8. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  9. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  10. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  11. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  12. 2023Large Language Models as Analogical Reasonerscited by 107
6 more
  1. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  2. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  3. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  4. 2023RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbationscited by 41
  5. 2023The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Pythoncited by 29
  6. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?cited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Capabilities and Invariability of Large Language Models
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
  3. 2025Method-Based Reasoning for Large Language Models: Extraction, Reuse, and Continuous Improvement
  4. 2024General Purpose Verification for Chain of Thought Prompting
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2024A NotSo Simple Way to Beat Simple Bench
  7. 2022Towards Reasoning in Large Language Models: A Survey
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance