# How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling

arXiv: https://arxiv.org/abs/2604.04791 · HTML: https://mlsanity.com/paper/2604.04791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [Evaluating Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2512.15567/) (2025)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [OJBench: A Competition Level Code Benchmark For Large Language Models](https://mlsanity.com/paper/2506.16395/) (2025)
- [AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators](https://mlsanity.com/paper/2508.09101/) (2025)
- [MM-Agent: LLM as Agents for Real-world Mathematical Modeling Problem](https://mlsanity.com/paper/2505.14148/) (2025)
- [Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agents](https://mlsanity.com/paper/2505.09970/) (2025)
- [Unified Software Engineering Agent as AI Software Engineer](https://mlsanity.com/paper/2506.14683/) (2025)
- [EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving](https://mlsanity.com/paper/2509.17677/) (2025)
- [SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning](https://mlsanity.com/paper/2511.08151/) (2025)
- [ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges](https://mlsanity.com/paper/2505.15068/) (2025)
- [ARISE: Agentic Rubric-Guided Iterative Survey Engine for Automated Scholarly Paper Generation](https://mlsanity.com/paper/2511.17689/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have achieved strong performance on reasoning benchmarks, yet their ability to solve real-world problems requiring end-to-end workflows remains unclear. Mathematical modeling competitions provide a stringent testbed for evaluating such end-to-end problem-solving capability. We propose a problem-oriented, stage-wise evaluation framework that assesses LLM performance across modeling stages using expert-verified criteria. We validate the framework's reliability by comparing automatic scores with independent human expert judgments on problems from the China Postgraduate Mathematical Contest in Modeling, demonstrating substantially stronger alignment than existing evaluation schemes. Using this framework, we reveal a comprehension-execution gap in state-of-the-art LLMs: while they perform well in early stages such as problem identification and formulation, they exhibit persistent deficiencies in execution-oriented stages including model solving, code implementation, and result analysis. These gaps persist even with increased model scale. We further trace these failures to insufficient specification, missing verification, and lack of validation, with errors propagating across stages without correction. Our findings suggest that bridging this gap requires approaches beyond model scaling, offering insights for applying LLMs to complex real-world problem solving.