# Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis

arXiv: https://arxiv.org/abs/2606.21972 · HTML: https://mlsanity.com/paper/2606.21972/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://mlsanity.com/paper/2506.06941/) (2025)
- [MathArena: Evaluating LLMs on Uncontaminated Math Competitions](https://mlsanity.com/paper/2505.23281/) (2025)
- [Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs](https://mlsanity.com/paper/2505.00127/) (2025)
- [The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofs](https://mlsanity.com/paper/2506.21621/) (2025)
- [Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens](https://mlsanity.com/paper/2602.13517/) (2026)

## Built on by (most cited first, 1 of 1)

- [An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks](https://mlsanity.com/paper/2609.18052/) (2026)

## Abstract (arXiv)

We study how the effort and success probability of frontier AI systems scale with human difficulty on problems from Project Euler, an online platform of computational mathematics problems. Our dataset, from the MathArena benchmark, consists of 3840 attempts across 50 problems and 26 model configurations, with problem difficulty measured by the site's public human solve times. Motivated by a proposal of Timothy Gowers, we test a power-law relation tmachine = a · thumanb between generated-token cost per successful answer and human time, and find b < 1 for 20 of the 25 models with usable fits, including the strongest base models; this operationalization therefore does not support an earlier prediction that machines scale worse than humans with difficulty. We also investigate whether success probability on the tested problems can be modeled by a simple exponential decay psuccess = e^c thuman, predicting a linear relation between log psuccess and thuman. Using a binning approach for data aggregation we find moderate empirical support (median bin-level R2 = 0.92 across the 22 best-covered configurations) for this model. Following METR, we also fit logistic success curves and extract 50% task-length horizons h50; the strongest configurations in our 20 April 2026 snapshot reach roughly 2.5--4.3 hours on our fastest-five human baseline, with a log-linear fit through the state-of-the-art frontier giving a descriptive doubling time of about 75~days for the SOTA h50.