TCSAlgBench: Benchmarking Automated Proving for Research-Level Theoretical Computer Science
Not on the map. Why?
Abstract (arXiv)
Large language models perform strongly on competition mathematics, but their research-level reasoning remains difficult to evaluate systematically. Theoretical computer science (TCS) connects algorithm design to explicit guarantees and fundamental limits, providing a setting for evaluating whether models can justify computational improvements with arguments humans can inspect. We introduce TCSAlgBench, a benchmark and reusable pipeline for natural-language proof discovery, comprising 398 theorem-level challenges from 138 STOC and COLT 2026 papers. Expert-designed rules complete paper-specific context, preserve computational assumptions and quantitative guarantees, and withhold constructions when discovering an algorithm is part of the task. For each task, prover systems receive theorem statements and access to cited prior work. The pipeline supports fresh, versioned challenge batches from newly released papers. We evaluate ten model configurations from four families under direct inference and prover-verifier discussion, and compare four agent workflows under matched model-call opportunities. All evaluations use the full benchmark. In the model comparison, GPT-5.6 Sol max achieves the highest five-run verifier-accepted coverage at 23.6% after 10-round discussion. Discussion and repeated sampling improve coverage. In the separate agent comparison using GPT-5.5 xhigh, decomposition improves coverage over discussion, and agentic planning achieves the highest five-run verifier-accepted coverage at 25.4%. TCSAlgBench provides a refreshable testbed for measuring progress in model reasoning and studying how agent workflows support research-level proof discovery.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 63 of its 70 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
63 of 70 references.
References · 63
- 2021Training Verifiers to Solve Math Word Problemscited by 9,916
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
- 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 819
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 547
- 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 420
- 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 383
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 302
- 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 285
- 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 281
- 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 280
- 2022Autoformalization with Large Language Modelscited by 259
38 more
- 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 239
- 2022HyperTree Proof Search for Neural Theorem Provingcited by 195
- 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 194
- 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 185
- 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 155
- 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 125
- 2024Lean Workbook: A large-scale Lean problem set formalized from natural language math problemscited by 117
- 2023Cumulative Reasoning with Large Language Modelscited by 94
- 2023An In-Context Learning Agent for Formal Theorem-Provingcited by 80
- 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 80
- 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 69
- 2025Hilbert: Recursively Building Formal Proofs with Informal Reasoningcited by 57
- 2026Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematicscited by 45
- 2026Towards Autonomous Mathematics Researchcited by 37
- 2026Agents' Last Examcited by 29
- 2026First Proofcited by 28
- 2025The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofscited by 27
- 2025Scaling up Multi-Turn Off-Policy RL and Multi-Agent Tree Search for LLM Step-Proverscited by 27
- 2024Proving Theorems Recursivelycited by 26
- 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 26
- 2026Advancing Mathematics Research with AI-Driven Formal Proof Searchcited by 21
- 2025Reliable Fine-Grained Evaluation of Natural Language Math Proofscited by 14
- 2026A Minimal Agent for Automated Theorem Provingcited by 14
- 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 12
- 2026QED: An Open-Source Multi-Agent System for Generating Mathematical Proofs on Open Problemscited by 12
- 2026LeanMarathon: Toward Reliable AI Co-Mathematicians through Long-Horizon Lean Autoformalizationcited by 12
- 2026LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworkscited by 11
- 2023A New Approach Towards Autoformalizationcited by 10
- 2026LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematicscited by 8
- 2026RMA: an Agentic System for Research-Level Mathematical Problemscited by 7
- 2026Formalizing Mathematics at Scalecited by 7
- 2026Goedel-Architect: Streamlining Formal Theorem Proving with Blueprint Generation and Refinementcited by 7
- 2026Danus: Orchestrating Mathematical Reasoning Agents with Fact-Graph Memorycited by 7
- 2026ProofCouncil: An LLM Agent for Solving Open Mathematical Problemscited by 7
- 2026First Proof Second Batchcited by 6
- 2026Calibeating Made Simplecited by 5
- 2026Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalizationcited by 5
- 2026TheoremGraph: Bridging Formal and Informal Mathematicscited by 4
50 most cited of 63 All 63 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science
- 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
- 2026LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
- 2025Scaling Natural-Language Graph-Based Test Time Compute for Automated Theorem Proving
- 2026TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
- 2026Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification
- 2026Direct Optimization of Generators for Search in Automated Theorem Proving
- 2025ProofCompass: Enhancing Specialized Provers with LLM Guidance