# MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling

arXiv: https://arxiv.org/abs/2605.12154 · HTML: https://mlsanity.com/paper/2605.12154/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 76 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 21 of 21)

- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine](https://mlsanity.com/paper/2407.08739/) (2024)
- [OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models](https://mlsanity.com/paper/2402.10172/) (2024)
- [MapQA: A Dataset for Question Answering on Choropleth Maps](https://mlsanity.com/paper/2211.08545/) (2022)
- [MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models](https://mlsanity.com/paper/2409.00147/) (2024)
- [CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2401.11944/) (2024)
- [LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages](https://mlsanity.com/paper/2405.13144/) (2024)
- [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](https://mlsanity.com/paper/2505.11792/) (2025)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision](https://mlsanity.com/paper/2505.13427/) (2025)
- [FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow](https://mlsanity.com/paper/2505.17399/) (2025)
- [Holy Grail 2.0: From Natural Language to Constraint Models](https://mlsanity.com/paper/2308.01589/) (2023)
- [MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model](https://mlsanity.com/paper/2409.13729/) (2024)
- [Constructing Industrial-Scale Optimization Modeling Benchmark](https://mlsanity.com/paper/2602.10450/) (2026)
- [ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling](https://mlsanity.com/paper/2510.27610/) (2025)
- [PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documents](https://mlsanity.com/paper/2406.13923/) (2024)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [ORThought: Benchmarking and Automating Logistics Optimization Modeling](https://mlsanity.com/paper/2508.14410/) (2025)

## Built on by (most cited first, 2 of 2)

- [ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making](https://mlsanity.com/paper/2608.07584/) (2026)
- [Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling](https://mlsanity.com/paper/2609.38884/) (2026)

## Abstract (arXiv)

Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but instance information is conveyed through visual artifacts such as tables, graphs, maps, schedules, and dashboards. We introduce multimodal optimization modeling, a benchmark setting in which models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each with an exact solver, and builds both the model-facing inputs and hidden reference files from the same verified source. We instantiate the framework as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose models and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The results show that the task remains far from solved: the best two models reach 52.1% and 51.3% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4% on easy instances and 15.9% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Failure attribution shows that errors arise both when extracting instance data from text and visuals and when turning extracted data into solver-correct formulations and code. MM-OptBench provides a testbed for solver-grounded, decision-oriented multimodal intelligence.