mlsanityData to 7 Oct 2026

OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

arXiv 2407.09887 ↗First posted 13 Jul 2024cs.LG math.OCcited by 66

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations. In this work, we propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs. OptiBench contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs' solving ability. In our benchmark, LLMs are required to call a code solver to provide precise numerical answers. Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ReSocratic. Unlike general data synthesis methods that proceed from questions to answers, ReSocratic first incrementally synthesizes formatted optimization demonstration with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ReSocratic-29k dataset. We further conduct supervised fine-tuning with ReSocratic-29k on multiple open-source models. Experimental results show that ReSocratic-29k significantly improves the performance of open-source models.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 30 of its 51 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

30 of 51 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
18 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  3. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 133
  4. 2024OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Modelscited by 105
  5. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  6. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 58
  7. 2022Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptionscited by 54
  8. 2024Process-Driven Autoformalization in Lean 4cited by 45
  9. 2023Prompting Large Language Models With the Socratic Methodcited by 43
  10. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  11. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 42
  12. 2022LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learningcited by 24
  13. 2023TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Modelscited by 24
  14. 2023The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Modelscited by 22
  15. 2023Large Language Model for Science: A Study on P vs. NPcited by 12
  16. 2023Data Augmentation for Code Translation with Comparable Corpora and Multiple Referencescited by 9
  17. 2023Exploring Data Augmentation for Code Generation Taskscited by 7
  18. 2023CLOMO: Counterfactual Logical Modification with Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 20; 2026: 45.

66 papers.

Sort:

By area:

  1. 2025TreeRPO: Tree Relative Policy Optimizationcited by 45
  2. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modelingcited by 44
  3. 2024OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scalecited by 37
  4. 2025A Survey of Optimization Modeling Meets LLMs: Progress and Future Directionscited by 30
  5. 2025CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimizationcited by 28
  6. 2026Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoningcited by 23
  7. 2025OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learningcited by 16
  8. 2025Auto-Formulating Dynamic Programming Problems with Large Language Modelscited by 15
  9. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  10. 2025AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Librarycited by 12
  11. 2025ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Researchcited by 10
  12. 2026Constructing Industrial-Scale Optimization Modeling Benchmarkcited by 10
38 more
  1. 2025EquivaMap: Leveraging LLMs for Automatic Equivalence Checking of Optimization Formulationscited by 9
  2. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  3. 2025OptiMind: Teaching LLMs to Think Like Optimization Expertscited by 9
  4. 2025OptiChat: Bridging Optimization Models and Practitioners with Large Language Modelscited by 8
  5. 2025SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modelingcited by 8
  6. 2025ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modelingcited by 7
  7. 2025Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generationcited by 5
  8. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
  9. 2026NEMO: Execution-Aware Optimization Modeling via Autonomous Coding Agentscited by 5
  10. 2026Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verificationcited by 5
  11. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  12. 2026OptiRepair: Closed-Loop Diagnosis and Repair of Supply Chain Optimization Models with LLM Agentscited by 4
  13. 2026Strategy-Aware Optimization Modeling with Reasoning LLMscited by 4
  14. 2026FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimizationcited by 4
  15. 2026OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solvingcited by 3
  16. 2025When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuningcited by 2
  17. 2026NC2C: Automated Convexification of Generic Non-Convex Optimization Problemscited by 2
  18. 2026MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Researchcited by 2
  19. 2026Execution-Verified Reinforcement Learning for Optimization Modelingcited by 2
  20. 2026Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimizationcited by 2
  21. 2026TriVAL: A Tri-Validation Framework for Faithful Automatic Optimization Modelingcited by 2
  22. 2026OptSkills: Learning Generalizable Optimization Skills from Problem Archetypes via Cluster-Based Distillationcited by 2
  23. 2026Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problemscited by 2
  24. 2025Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimizationcited by 1
  25. 2025DAOpt: Modeling and Evaluation of Data-Driven Optimization under Uncertainty with LLMscited by 1
  26. 2026Contract2Plan: Verified Contract-Grounded Retrieval-Augmented Optimization for BOM-Aware Procurement and Multi-Echelon Inventory Planningcited by 1
  27. 2026Grammar-Aware Literate Generative Mathematical Programming with Compiler-in-the-Loopcited by 1
  28. 2026Automated Reformulation of Robust Optimization via Memory-Augmented Large Language Modelscited by 1
  29. 2026MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resourcescited by 1
  30. 2026Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limitscited by 1
  31. 2026PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Languagecited by 1
  32. 2026SCHEDBench: A Benchmark for Evaluating LLM Constraint Faithfulness in Natural-Language Combinatorial Schedulingcited by 1
  33. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  34. 2025Step-Opt: Boosting Optimization Modeling in LLMs through Iterative Data Synthesis and Structured Validationcited by 0
  35. 2026DALI: LLM-Agent Enhanced Dual-Stream Adaptive Leadership Identification for Group Recommendationscited by 0
  36. 2026NED-Tree: Bridging the Semantic Gap with Nonlinear Element Decomposition Tree for LLM Nonlinear Optimization Modelingcited by 0
  37. 2026Co-evolving Agent Architectures and Interpretable Reasoning for Automated Optimizationcited by 0
  38. 2026Dual-Cluster Memory Agent: Resolving Multi-Paradigm Ambiguity in Optimization Problem Solvingcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling
  2. 2025Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation
  3. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  4. 2026MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
  5. 2024LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch
  6. 2026ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization
  7. 2026OptiSkill: A Hierarchical and Evolving SkillBank for LLM-Based Optimization Modeling
  8. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces