# LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch

arXiv: https://arxiv.org/abs/2410.13213 · HTML: https://mlsanity.com/paper/2410.13213/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models](https://mlsanity.com/paper/2402.10172/) (2024)
- [Large Language Models for Supply Chain Optimization](https://mlsanity.com/paper/2307.03875/) (2023)
- [LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages](https://mlsanity.com/paper/2405.13144/) (2024)
- [LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation](https://mlsanity.com/paper/2403.01131/) (2024)
- [Large Language Model for Multi-objective Evolutionary Optimization](https://mlsanity.com/paper/2310.12541/) (2023)
- [InstOptima: Evolutionary Multi-objective Instruction Optimization via Large Language Model-based Instruction Operators](https://mlsanity.com/paper/2310.17630/) (2023)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)

## Built on by (most cited first, 30 of 102)

- [Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling](https://mlsanity.com/paper/2505.11792/) (2025)
- [A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions](https://mlsanity.com/paper/2508.10047/) (2025)
- [Large Language Models as End-to-end Combinatorial Optimization Solvers](https://mlsanity.com/paper/2509.16865/) (2025)
- [StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models](https://mlsanity.com/paper/2509.22558/) (2025)
- [HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization](https://mlsanity.com/paper/2506.07972/) (2025)
- [OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learning](https://mlsanity.com/paper/2511.09092/) (2025)
- [AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Library](https://mlsanity.com/paper/2510.18428/) (2025)
- [Automated Optimization Modeling via a Localizable Error-Driven Perspective](https://mlsanity.com/paper/2602.11164/) (2026)
- [Constructing Industrial-Scale Optimization Modeling Benchmark](https://mlsanity.com/paper/2602.10450/) (2026)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [OptiMind: Teaching LLMs to Think Like Optimization Experts](https://mlsanity.com/paper/2509.22979/) (2025)
- [ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming](https://mlsanity.com/paper/2510.05774/) (2025)
- [SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search](https://mlsanity.com/paper/2510.16916/) (2025)
- [ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling](https://mlsanity.com/paper/2510.27610/) (2025)
- [MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization](https://mlsanity.com/paper/2505.17745/) (2025)
- [EvoCut: Strengthening Integer Programs via Evolution-Guided Language Models](https://mlsanity.com/paper/2508.11850/) (2025)
- [SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling](https://mlsanity.com/paper/2510.05115/) (2025)
- [ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization](https://mlsanity.com/paper/2602.15983/) (2026)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization](https://mlsanity.com/paper/2605.25246/) (2026)
- [DesignX: Human-Competitive Algorithm Designer for Black-Box Optimization](https://mlsanity.com/paper/2505.17866/) (2025)
- [Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generation](https://mlsanity.com/paper/2508.03117/) (2025)
- [HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimization](https://mlsanity.com/paper/2511.19669/) (2025)
- [NEMO: Execution-Aware Optimization Modeling via Autonomous Coding Agents](https://mlsanity.com/paper/2601.21372/) (2026)
- [Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification](https://mlsanity.com/paper/2605.29556/) (2026)
- [Strategy-Aware Optimization Modeling with Reasoning LLMs](https://mlsanity.com/paper/2605.02545/) (2026)
- [Performance of LLMs on Stochastic Modeling Operations Research Problems: From Theory to Practice](https://mlsanity.com/paper/2506.23924/) (2025)
- [CALM Before the STORM: Unlocking Native Reasoning for Optimization Modeling](https://mlsanity.com/paper/2510.04204/) (2025)

## Abstract (arXiv)

Optimization problems are prevalent across various scenarios. Formulating and then solving optimization problems described by natural language often requires highly specialized human expertise, which could block the widespread application of optimization-based decision making. To automate problem formulation and solving, leveraging large language models (LLMs) has emerged as a potential way. However, this kind of approach suffers from the issue of optimization generalization. Namely, the accuracy of most current LLM-based methods and the generality of optimization problem types that they can model are still limited. In this paper, we propose a unified learning-based framework called LLMOPT to boost optimization generalization. Starting from the natural language descriptions of optimization problems and a pre-trained LLM, LLMOPT constructs the introduced five-element formulation as a universal model for learning to define diverse optimization problem types. Then, LLMOPT employs the multi-instruction tuning to enhance both problem formalization and solver code generation accuracy and generality. After that, to prevent hallucinations in LLMs, such as sacrificing solving accuracy to avoid execution errors, the model alignment and self-correction mechanism are adopted in LLMOPT. We evaluate the optimization generalization ability of LLMOPT and compared methods across six real-world datasets covering roughly 20 fields such as health, environment, energy and manufacturing, etc. Extensive experiment results show that LLMOPT is able to model various optimization problem types such as linear/nonlinear programming, mixed integer programming, and combinatorial optimization, and achieves a notable 11.08% average solving accuracy improvement compared with the state-of-the-art methods. The code is available at https://github.com/caigaojiang/LLMOPT.