mlsanityData to 14 Oct 2026

The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models

arXiv 2511.02589 ↗First posted 4 Nov 2025cs.AIcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We present ORCA (Omni Research on Calculation in AI) Benchmark - a novel benchmark that evaluates large language models (LLMs) on multi-domain, real-life quantitative reasoning using verified outputs from Omni's calculator engine. In 500 natural-language tasks across domains such as finance, physics, health, and statistics, the five state-of-the-art systems (ChatGPT-5, Gemini~2.5~Flash, Claude~Sonnet~4.5, Grok~4, and DeepSeek~V3.2) achieved only 45--63,% accuracy, with errors mainly related to rounding (35,%) and calculation mistakes (33,%). Results in specific domains indicate strengths in mathematics and engineering, but weaknesses in physics and natural sciences. Correlation analysis (r ≈ 0.40--0.65) shows that the models often fail together but differ in the types of errors they make, highlighting their partial complementarity rather than redundancy. Unlike standard math datasets, ORCA evaluates step-by-step reasoning, numerical precision, and domain generalization across real problems from finance, physics, health, and statistics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 24 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

14 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2022PAL: Program-aided Language Modelscited by 652
  5. 2025Humanity's Last Examcited by 491
  6. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 391
  7. 2023Mathematical Capabilities of ChatGPTcited by 268
  8. 2023Challenges and Applications of Large Language Modelscited by 254
  9. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  10. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  11. 2025IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generationcited by 13
  12. 2025Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarkscited by 9
1 more
  1. 2023ChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questionscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
  2. 2023ARB: Advanced Reasoning Benchmark for Large Language Models
  3. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  4. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  5. 2024Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
  6. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  7. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  8. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models