ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
We present ThinkProbe, a framework for structural analysis of LLM reasoning traces. ThinkProbe converts each trace into a Thought Graph a directed graph with cycles, 8 node types, and 6 edge types and derives a 19-metric five-dimensional cognitive profile (5D-CP: Breadth, Depth, Structure, Metacognitive, Efficiency) through a fully non-generative pipeline combining rule-based segmentation and discriminative semantic linking. Applied to 4{,}200 traces from 7 native reasoning models across 200 open-ended questions and 10 cognitive domains, ThinkProbe reveals that reasoning structure is a stable, model-level property: between-model variance exceeds between-domain variance by up to fourfold across four of five cognitive dimensions, with Structure showing genuine sensitivity to question domain, exposing qualitatively distinct cognitive profiles invisible to accuracy-based evaluation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 14 of its 33 references that are in the map. First posted on arXiv 27 Jun 2026.
Builds on
14 of 33 references.
Closest ancestors · 6
Same direction, newest first
- 2026CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs7 Mar
- 2025Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinking9 Oct
- 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMs18 Aug
- 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
- 2025ReasoningFlow: Semantic Structure of Complex Reasoning Traces3 Jun
- 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models28 May
Other references · 8
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
- 2025Phi-4-reasoning Technical Reportcited by 146
- 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
- 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
- 2025Schoenfeld's Anatomy of Mathematical Reasoning by Language Modelscited by 8
- 2026Diagnosing Pathological Chain-of-Thought in Reasoning Modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Grounding LLM Reasoning with Knowledge Graphs
- 2024Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization
- 2025Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?
- 2026ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces
- 2026Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
- 2026Cognitive Profiling of LRMs' Reasoning Traces Using Bloom's Taxonomy
- 2026Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
- 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM