mlsanityData to 13 Oct 2026

CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning

arXiv 2510.27094 ↗First posted 31 Oct 2025cs.AIcited by 2

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

State-of-the-art (SOTA) LLMs have progressed from struggling on proof-based Olympiad problems to solving most of the IMO 2025 problems, with leading systems reportedly handling 5 of 6 problems. Given this progress, we assess how well these models can grade proofs: detecting errors, judging their severity, and assigning fair scores beyond binary correctness. We study proof-analysis capabilities using a corpus of 90 Gemini 2.5 Pro-generated solutions that we grade on a 1-4 scale with detailed error annotations, and on MathArena solution sets for IMO/USAMO 2025 scored on a 0-7 scale. Our analysis shows that models can reliably flag incorrect (including subtly incorrect) solutions but exhibit calibration gaps in how partial credit is assigned. To address this, we introduce agentic workflows that extract and analyze reference solutions and automatically derive problem-specific rubrics for a multi-step grading process. We instantiate and compare different design choices for the grading workflows, and evaluate their trade-offs. Across our annotated corpus and MathArena, our proposed workflows achieve higher agreement with human grades and more consistent handling of partial credit across metrics. We release all code, data, and prompts/logs to facilitate future research.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 4 of its 22 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

4 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024HARP: A challenging human-annotated math reasoning benchmark11 Dec

Other references · 3

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  2. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 225
  3. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025RefGrader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows
  2. 2026ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics
  3. 2025RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning
  4. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics
  5. 2026AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
  6. 2025Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
  7. 2025Towards Robust Mathematical Reasoning
  8. 2025EEFSUVA: A New Mathematical Olympiad Benchmark