LeanCat: A Benchmark Suite for Formal Category Theory in Lean (Part I: 1-Categories)
Not on the map. Why?
Abstract (arXiv)
While large language models (LLMs) have demonstrated impressive capabilities in formal theorem proving, current benchmarks fail to adequately measure library-grounded abstraction -- the ability to reason with high-level interfaces and reusable structures central to modern mathematics and software engineering. We introduce LeanCat, a challenging benchmark comprising 100 fully formalized category-theory tasks in Lean. Unlike algebra or arithmetic, category theory serves as a rigorous stress test for structural, interface-level reasoning. Our evaluation reveals a severe abstraction gap: the best state-of-the-art model solves only 12.0% of tasks at pass@4, with performance collapsing from 55.0% on Easy tasks to 0.0% on High-difficulty tasks, highlighting a failure in compositional generalization. To overcome this, we evaluate LeanBridge, a retrieval-augmented agent that employs a retrieve-generate-verify loop. LeanBridge achieves a peak success rate of 24.0% -- doubling the performance of the best static baseline. These results empirically demonstrate that iterative refinement and dynamic library retrieval are not merely optimizations but strict necessities for neuro-symbolic reasoning in abstract domains. LeanCat offers a compact, reusable testbed for tracking progress toward reliable, research-level formalization.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 31 Dec 2025.
Builds on
13 of 27 references.
References · 13
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
- 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
- 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 173
- 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 153
- 2022Formal Mathematics Statement Curriculum Learningcited by 141
- 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
- 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 54
- 2025Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experiencecited by 36
- 2025FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levelscited by 23
- 2025LeanExplore: A search engine for Lean 4 declarationscited by 13
1 more
- 2025StepFun-Prover Preview: Let's Think and Verify Step by Stepcited by 12
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026A Minimal Agent for Automated Theorem Provingcited by 13
- 2026SorryDB: Can AI Provers Complete Real-World Lean Theorems?cited by 4
- 2026Theory-Scale Auto-Formalization of Logics for Computer Sciencecited by 3
- 2026MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysiscited by 1
- 2026Theory-Scale Auto-Formalization of Logics for Computer Science25 Jun 2026
- 2026MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis11 Jun 2026
- 2026SorryDB: Can AI Provers Complete Real-World Lean Theorems?3 Mar 2026
- 2026A Minimal Agent for Automated Theorem Proving27 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
- 2025SITA: A Framework for Structure-to-Instance Theorem Autoformalization
- 2025DRIFT: Decompose, Retrieve, Illustrate, then Formalize Theorems
- 2026CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean
- 2026Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis
- 2026Construction-Verification: A Benchmark for Applied Mathematics in Lean 4
- 2024LeanReasoner: Boosting Complex Logical Reasoning with Lean
- 2026LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks