Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game
On the map: LLM reasoning → Mathematical reasoning
Abstract (arXiv)
While Large Language Models have achieved notable success on formal mathematics benchmarks such as MiniF2F, it remains unclear whether these results stem from genuine logical reasoning or semantic pattern matching against pre-training data. This paper identifies Architectural Reasoning: the ability to synthesize formal proofs using exclusively local axioms and definitions within an alien math domain, as the necessary ability for future automated theorem discovery AI. We use the Obfuscated Natural Number Game, a benchmark to evaluate Architectural Reasoning. By renaming identifiers in the Natural Number Game in Lean 4, we created a zero-knowledge, closed environment. We evaluate state-of-the-art models, finding a universal latency tax where obfuscation increases inference time. The results also reveal a divergence in robustness: while general models (Claude-Sonnet-4.5, GPT-4o) suffer performance degradation, reasoning models (DeepSeek-R1, GPT-5, DeepSeek-Prover-V2) maintain the same accuracy despite the absence of semantic cues. These findings provide a quantitative metric for assessing the true capacity for mathematical reasoning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 5 of its 14 references that are in the map. First posted on arXiv 1 May 2026.
Builds on
5 of 14 references.
Closest ancestors · 1
Same direction, newest first
Other references · 4
- 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
- 2025Early science acceleration experiments with GPT-5cited by 35
- 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Modelscited by 11
- 2024SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specificationscited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Advancing Mathematics Research with AI-Driven Formal Proof Search
- 2025From Informal to Formal -- Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs
- 2026Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
- 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoning
- 2025FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
- 2025HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs
- 2025DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learning
- 2026Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery