mlsanityData to 7 Oct 2026

LeanCat: A Benchmark Suite for Formal Category Theory in Lean (Part I: 1-Categories)

arXiv 2512.24796 ↗First posted 31 Dec 2025cs.LO cs.AI cs.FLcited by 4

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) have demonstrated impressive capabilities in formal theorem proving, current benchmarks fail to adequately measure library-grounded abstraction -- the ability to reason with high-level interfaces and reusable structures central to modern mathematics and software engineering. We introduce LeanCat, a challenging benchmark comprising 100 fully formalized category-theory tasks in Lean. Unlike algebra or arithmetic, category theory serves as a rigorous stress test for structural, interface-level reasoning. Our evaluation reveals a severe abstraction gap: the best state-of-the-art model solves only 12.0% of tasks at pass@4, with performance collapsing from 55.0% on Easy tasks to 0.0% on High-difficulty tasks, highlighting a failure in compositional generalization. To overcome this, we evaluate LeanBridge, a retrieval-augmented agent that employs a retrieve-generate-verify loop. LeanBridge achieves a peak success rate of 24.0% -- doubling the performance of the best static baseline. These results empirically demonstrate that iterative refinement and dynamic library retrieval are not merely optimizations but strict necessities for neuro-symbolic reasoning in abstract domains. LeanCat offers a compact, reusable testbed for tracking progress toward reliable, research-level formalization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

13 of 27 references.

References · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  3. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  4. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  5. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 173
  6. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 153
  7. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  8. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  9. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 54
  10. 2025Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experiencecited by 36
  11. 2025FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levelscited by 23
  12. 2025LeanExplore: A search engine for Lean 4 declarationscited by 13
1 more
  1. 2025StepFun-Prover Preview: Let's Think and Verify Step by Stepcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
  2. 2025SITA: A Framework for Structure-to-Instance Theorem Autoformalization
  3. 2025DRIFT: Decompose, Retrieve, Illustrate, then Formalize Theorems
  4. 2026CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean
  5. 2026Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis
  6. 2026Construction-Verification: A Benchmark for Applied Mathematics in Lean 4
  7. 2024LeanReasoner: Boosting Complex Logical Reasoning with Lean
  8. 2026LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks