mlsanityData to 7 Oct 2026

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

arXiv 2609.32235 ↗First posted 26 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 72 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

54 of 72 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking8 Jan
  2. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI7 Nov
  3. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models10 Oct
  4. 2024Give me a hint: Can LLMs take a hint to solve math problems?8 Oct
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models7 Oct
  6. 2024Not All LLM Reasoners Are Created Equal2 Oct
  7. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI18 Jun
  8. 2024Evaluating Mathematical Reasoning Beyond Accuracy8 Apr
  9. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers29 Feb
  10. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  11. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug
  12. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models20 Jul
1 more
  1. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models24 May

Other references · 41

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
29 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  13. 2024DeepSeek-V3 Technical Reportcited by 724
  14. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  15. 2022PAL: Program-aided Language Modelscited by 635
  16. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  17. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  18. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  19. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  20. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  21. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 153
  22. 2021The Benchmark Lotterycited by 133
  23. 2022Holistic Evaluation of Language Modelscited by 115
  24. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  25. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  26. 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
  27. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  28. 2024Optimizing Chain-of-Thought Reasoning: Tackling Arranging Bottleneck via Plan Augmentationcited by 2
  29. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  2. 2025Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
  3. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics
  4. 2024Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems
  5. 2024Large Language Models Struggle with Unreasonability in Math Problems
  6. 2024From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems
  7. 2025Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning
  8. 2025Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation