mlsanityData to 14 Oct 2026

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning

arXiv 2601.00791 ↗First posted 2 Jan 2026cs.LG cs.AI cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle. We show that valid mathematical reasoning induces a measurable, training-free spectral signature in transformer attention. By treating each attention matrix as a weighted token graph, we extract four diagnostics: Fiedler value, High-Frequency Energy Ratio (HFER), spectral entropy, and smoothness, that require no learned parameters. Experiments across seven models from four architectural families yield effect sizes up to Cohen's d = 3.30 (p < 10-116), enabling 85--96% single-threshold classification accuracy. Two findings sharpen the interpretation. First, Platonic validity: the spectral signal tracks logical coherence rather than compiler acceptance, proofs rejected for timeouts or missing imports are correctly classified as valid, a distinction confirmed by a manual audit (kappa = 0.82, n = 51). Second, architectural determinism: Sliding Window Attention shifts the discriminative feature from HFER to smoothness (d = 2.09, p < 10-48), showing that attention design governs which spectral channel encodes reasoning quality. Causal ablation confirms the signature traces induction-head circuits. The method generalises to informal chain-of-thought (d = 0.78, p < 10-3), and in proof search, HFER reranking improves Best-of-16 Pass@1 by +4.4--6.6%, matching 98% of the AUC of fully supervised probes with zero labels. Spectral graph analysis is a principled, architecture-aware primitive for reasoning verification.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 54 of its 77 references that are in the map. First posted on arXiv 2 Jan 2026.

Builds on

54 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May

Other references · 53

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
38 more
  1. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  5. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  6. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  10. 2019Designing and Interpreting Probes with Control Taskscited by 640
  11. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  12. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  13. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  14. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  15. 2017Graph Signal Processing: Overview, Challenges and Applicationscited by 413
  16. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  17. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  18. 2023Llemma: An Open Language Model For Mathematicscited by 388
  19. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  20. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  21. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 247
  22. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  23. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  24. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  25. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  26. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  27. 2023Specformer: Spectral Graph Neural Networks Meet Transformerscited by 102
  28. 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 97
  29. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  30. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiadcited by 74
  31. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  32. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  33. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  34. 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 60
  35. 2025APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoningcited by 35
  36. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematicscited by 31
  37. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
  38. 2025Towards Mechanistic Interpretability of Graph Transformers via Attention Graphscited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Spectral Geometry of Thought: Phase Transitions, Instruction Reversal, Token-Level Dynamics, and Perfect Correctness Prediction in How Transformers Reason
  2. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
  3. 2026The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models
  4. 2026Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
  5. 2026Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration
  6. 2026A Comprehensive Anatomy of Human and DeepSeek-R1 LLM Mathematical Reasoning
  7. 2026The Geometry of Logic: Stratification Induces Semantic Structure and Robust Reasoning
  8. 2026Unveiling the Reasoning Process of Large Language Models