mlsanityData to 14 Oct 2026

Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework

arXiv 2605.24661 ↗First posted 23 May 2026cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite remarkable progress on reasoning benchmarks, current LLM evaluation practice remains anchored to final-answer correctness, providing limited insight into how models reason, how reliably they behave under contextual variation, or how efficiently they reach conclusions. This paper proposes a unified multi-dimensional framework for measuring LLM reasoning quality from a behavioral perspective, operationalizing six theoretically grounded dimensions rooted in cognitive science: Correctness (CQ), Consistency (CS), Robustness (RS), Local Logical Coherence (LS), Efficiency (ES), and Stability (SS). The framework introduces deployment-aware aggregation, enabling context-specific model selection beyond accuracy-based leaderboards. Experiments across multiple LLMs and benchmarks reveal behaviors systematically concealed by single-metric evaluation, including the orthogonality of local logical coherence and correctness, deployment-context-dependent ranking inversions, and non-trivial dimensional profiles in small locally-deployed models. Discriminant validity analysis confirms that the proposed dimensions capture largely non-redundant signals. The resulting pipeline provides a foundation for diagnosing LLM reasoning behavior across deployment contexts, with domain-specific validation as a direction for future work.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 35 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

10 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey2 Apr
  2. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samples8 Nov

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  4. 2022Holistic Evaluation of Language Modelscited by 131
  5. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  6. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  7. 2024Robustness of Large Language Models to Perturbations in Textcited by 11
  8. 2026Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languagescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Do Large Language Models Know What They Don't Know II? A Fully Behavioral, Non-Cognitive Measure of Epistemic Honestycited by 2
  2. 2026When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  2. 2025What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
  3. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs
  4. 2026Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
  5. 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM
  6. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
  7. 2025Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It
  8. 2025Exploring LLM Reasoning Through Controlled Prompt Variations