Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle. We show that valid mathematical reasoning induces a measurable, training-free spectral signature in transformer attention. By treating each attention matrix as a weighted token graph, we extract four diagnostics: Fiedler value, High-Frequency Energy Ratio (HFER), spectral entropy, and smoothness, that require no learned parameters. Experiments across seven models from four architectural families yield effect sizes up to Cohen's d = 3.30 (p < 10-116), enabling 85--96% single-threshold classification accuracy. Two findings sharpen the interpretation. First, Platonic validity: the spectral signal tracks logical coherence rather than compiler acceptance, proofs rejected for timeouts or missing imports are correctly classified as valid, a distinction confirmed by a manual audit (kappa = 0.82, n = 51). Second, architectural determinism: Sliding Window Attention shifts the discriminative feature from HFER to smoothness (d = 2.09, p < 10-48), showing that attention design governs which spectral channel encodes reasoning quality. Causal ablation confirms the signature traces induction-head circuits. The method generalises to informal chain-of-thought (d = 0.78, p < 10-3), and in proof search, HFER reranking improves Best-of-16 Pass@1 by +4.4--6.6%, matching 98% of the AUC of fully supervised probes with zero labels. Spectral graph analysis is a principled, architecture-aware primitive for reasoning verification.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 54 of its 77 references that are in the map. First posted on arXiv 2 Jan 2026.
Builds on
54 of 77 references.
Closest ancestors · 1
Same direction, newest first
- 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
Other references · 53
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2023Let's Verify Step by Stepcited by 3,789
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
- 2019Language Models as Knowledge Bases?cited by 1,936
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
- 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
38 more
- 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2022Solving math word problems with process- and outcome-based feedbackcited by 901
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
- 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
- 2017Graph Signal Processing: Overview, Challenges and Applicationscited by 413
- 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
- 2020Generative Language Modeling for Automated Theorem Provingcited by 388
- 2023Llemma: An Open Language Model For Mathematicscited by 388
- 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
- 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
- 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
- 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
- 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
- 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
- 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
- 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 97
- 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
- 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
- 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
- 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
- 2017Classification and Geometry of General Perceptual Manifoldscited by 62
- 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 60
- 2025APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoningcited by 35
- 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
- 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
- 2025Towards Mechanistic Interpretability of Graph Transformers via Attention Graphscited by 13
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Spectral Archaeology: The Causal Topology of Model Evolutioncited by 0
- 2026Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Modelscited by 0
- 2026A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learningcited by 0
- 2026Attention-Graph Curvature for the Verification of Mathematical Reasoningcited by 0
- 2026Attention-Graph Curvature for the Verification of Mathematical Reasoning12 Oct 2026
- 2026A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning4 Aug 2026
- 2026Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Models17 Jun 2026
- 2026Spectral Archaeology: The Causal Topology of Model Evolution6 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2026The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
- 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
- 2026The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models
- 2026Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
- 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration
- 2026A Comprehensive Anatomy of Human and DeepSeek-R1 LLM Mathematical Reasoning
- 2026The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning
- 2026Unveiling the Reasoning Process of Large Language Models