# Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification

arXiv: https://arxiv.org/abs/2603.15617 · HTML: https://mlsanity.com/paper/2603.15617/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Mathematical exploration and discovery at scale](https://mlsanity.com/paper/2511.02864/) (2025)
- [Semi-Autonomous Mathematics Discovery with Gemini: A Case Study on the ErdHos Problems](https://mlsanity.com/paper/2601.22401/) (2026)
- [First Proof](https://mlsanity.com/paper/2602.05192/) (2026)
- [Accelerating Scientific Research with Gemini: Case Studies and Common Techniques](https://mlsanity.com/paper/2602.03837/) (2026)
- [Wait, Wait, Wait... Why Do Reasoning Models Loop?](https://mlsanity.com/paper/2512.12895/) (2025)
- [What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT](https://mlsanity.com/paper/2509.19284/) (2025)
- [FrontierCS: Evolving Challenges for Evolving Intelligence](https://mlsanity.com/paper/2512.15699/) (2025)
- [IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation](https://mlsanity.com/paper/2509.26076/) (2025)
- [Integer and Constraint Programming Revisited for Mutually Orthogonal Latin Squares](https://mlsanity.com/paper/2103.11018/) (2021)

## Built on by (most cited first, 9 of 9)

- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability](https://mlsanity.com/paper/2608.09538/) (2026)
- [OEIS Open: How many conjectures can language models turn into theorems?](https://mlsanity.com/paper/2608.11941/) (2026)
- [Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Science](https://mlsanity.com/paper/2603.28361/) (2026)
- [Autonomous Mathematical Discovery in an Open-World Multi-Agent Environment](https://mlsanity.com/paper/2608.23691/) (2026)
- [CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions](https://mlsanity.com/paper/2606.06526/) (2026)
- [ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics](https://mlsanity.com/paper/2606.10479/) (2026)
- [OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences](https://mlsanity.com/paper/2610.11118/) (2026)
- [AI Discovery in Combinatorics: From Problems to Solutions and Back to Bigger Problems](https://mlsanity.com/paper/2610.16555/) (2026)

## Abstract (arXiv)

Can AI make progress on important, unsolved mathematical problems? Large language models are now capable of sophisticated mathematical and scientific reasoning, but whether they can perform novel research is still widely debated and underexplored. We introduce HorizonMath, a benchmark of 113 predominantly unsolved problems spanning eight domains in mathematics and the mathematical sciences, paired with an open-source evaluation framework for automated verification. Our benchmark targets the generator-verifier gap: problems where discovery is hard and requires meaningful mathematical insight, but verification is computationally straightforward. This contrasts with most existing research-level benchmarks, which instead rely on formal proof verification or manual review, both of which are expensive to scale. Because these solutions are unknown, HorizonMath is resistant to data contamination, and most state-of-the-art models score under 10%. Using this framework, we identify six novel solutions to research problems that either resolve previously open questions or improve on the best-known published results, with GPT-5.4 Pro and GPT-5.6 Sol each discovering three of these solutions. Across seven frontier model families, reasoning efficiency and behavior also vary substantially. We release HorizonMath as an open challenge and a growing community resource, where each verified solution is a candidate contribution to the mathematical literature.