# LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems

arXiv: https://arxiv.org/abs/2403.01342 · HTML: https://mlsanity.com/paper/2403.01342/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [NEFTune: Noisy Embeddings Improve Instruction Finetuning](https://mlsanity.com/paper/2310.05914/) (2023)
- [Holy Grail 2.0: From Natural Language to Constraint Models](https://mlsanity.com/paper/2308.01589/) (2023)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)

## Built on by (most cited first, 18 of 18)

- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Large Language Models for Combinatorial Optimization: A Systematic Review](https://mlsanity.com/paper/2507.03637/) (2025)
- [A Survey on Mathematical Reasoning and Optimization with Large Language Models](https://mlsanity.com/paper/2503.17726/) (2025)
- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks](https://mlsanity.com/paper/2410.22296/) (2024)
- [EquivaMap: Leveraging LLMs for Automatic Equivalence Checking of Optimization Formulations](https://mlsanity.com/paper/2502.14760/) (2025)
- [EvoCut: Strengthening Integer Programs via Evolution-Guided Language Models](https://mlsanity.com/paper/2508.11850/) (2025)
- [CHORUS: Zero-shot Hierarchical Retrieval and Orchestration for Generating Linear Programming Code](https://mlsanity.com/paper/2505.01485/) (2025)
- [Structure-Aware Commitment Reduction for Network-Constrained Unit Commitment with Solver-Preserving Guarantees](https://mlsanity.com/paper/2604.02788/) (2026)
- [Large Language Model-Based Automatic Formulation for Stochastic Optimization Models](https://mlsanity.com/paper/2508.17200/) (2025)
- [Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems](https://mlsanity.com/paper/2510.10644/) (2025)
- [Large Language Model-Assisted Planning of Electric Vehicle Charging Infrastructure with Real-World Case Study](https://mlsanity.com/paper/2511.19055/) (2025)
- [StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modeling](https://mlsanity.com/paper/2606.15197/) (2026)
- [Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows](https://mlsanity.com/paper/2505.04354/) (2025)
- [LM4Opt-RA: A Multi-Candidate LLM Framework with Structured Ranking for Automating Network Resource Allocation](https://mlsanity.com/paper/2512.00039/) (2025)
- [HMACE: Heterogeneous Multi-Agent Collaborative Evolution for Combinatorial Optimization](https://mlsanity.com/paper/2605.07214/) (2026)
- [Generating Robust Portfolios of Optimization Models using Large Language Models](https://mlsanity.com/paper/2605.27013/) (2026)
- [Generative Artificial Intelligence (GenAI) to convert images of queuing networks into verifiable simulation models: an open-weight LLM workflow approach](https://mlsanity.com/paper/2607.24259/) (2026)

## Abstract (arXiv)

In the rapidly evolving field of natural language processing, the translation of linguistic descriptions into mathematical formulation of optimization problems presents a formidable challenge, demanding intricate understanding and processing capabilities from Large Language Models (LLMs). This study compares prominent LLMs, including GPT-3.5, GPT-4, and Llama-2-7b, in zero-shot and one-shot settings for this task. Our findings show GPT-4's superior performance, particularly in the one-shot scenario. A central part of this research is the introduction of `LM4OPT,' a progressive fine-tuning framework for Llama-2-7b that utilizes noisy embeddings and specialized datasets. However, this research highlights a notable gap in the contextual understanding capabilities of smaller models such as Llama-2-7b compared to larger counterparts, especially in processing lengthy and complex input contexts. Our empirical investigation, utilizing the NL4Opt dataset, unveils that GPT-4 surpasses the baseline performance established by previous research, achieving an F1-score of 0.63, solely based on the problem description in natural language, and without relying on any additional named entity information. GPT-3.5 follows closely, both outperforming the fine-tuned Llama-2-7b. These findings not only benchmark the current capabilities of LLMs in a novel application area but also lay the groundwork for future improvements in mathematical formulation of optimization problems from natural language input.