# Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

arXiv: https://arxiv.org/abs/2604.02666 · HTML: https://mlsanity.com/paper/2604.02666/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 20 of 20)

- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch](https://mlsanity.com/paper/2410.13213/) (2024)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling](https://mlsanity.com/paper/2407.09887/) (2024)
- [Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions](https://mlsanity.com/paper/2209.15565/) (2022)
- [Autoformulation of Mathematical Optimization Models Using LLMs](https://mlsanity.com/paper/2411.01679/) (2024)
- [Large Language Models Pass the Turing Test](https://mlsanity.com/paper/2503.23674/) (2025)
- [OptiMUS: Optimization Modeling Using MIP Solvers and large language models](https://mlsanity.com/paper/2310.06116/) (2023)
- [OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale](https://mlsanity.com/paper/2407.19633/) (2024)
- [JuMP 1.0: Recent improvements to a modeling language for mathematical optimization](https://mlsanity.com/paper/2206.03866/) (2022)
- ["I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programming](https://mlsanity.com/paper/2312.06908/) (2023)
- [AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Library](https://mlsanity.com/paper/2510.18428/) (2025)
- [Large-Scale Optimization Model Auto-Formulation: Harnessing LLM Flexibility via Structured Workflow](https://mlsanity.com/paper/2601.09635/) (2026)
- [A Better Match for Drivers and Riders: Reinforcement Learning at Lyft](https://mlsanity.com/paper/2310.13810/) (2023)
- [Decision Information Meets Large Language Models: The Future of Explainable Operations Research](https://mlsanity.com/paper/2502.09994/) (2025)
- [ORThought: Benchmarking and Automating Logistics Optimization Modeling](https://mlsanity.com/paper/2508.14410/) (2025)
- [Online Rack Placement in Large-Scale Data Centers: Online Sampling Optimization and Deployment](https://mlsanity.com/paper/2501.12725/) (2025)

## Built on by (most cited first, 3 of 3)

- [Democratizing Large-Scale Re-Optimization with LLM-Guided Model Patches](https://mlsanity.com/paper/2605.18692/) (2026)
- [FLARE: Verifying MILP Reformulations with LLM-Based Theorem Proving](https://mlsanity.com/paper/2608.25220/) (2026)
- [Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization](https://mlsanity.com/paper/2609.05258/) (2026)

## Abstract (arXiv)

Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations. We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker. We generate thousands of conversations in a school scheduling case study. Results show that one-shot evaluation is severely limiting: the same optimization agent converges to much higher-quality solutions through conversations. Then, this paper uses this methodology to demonstrate that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions, as compared to general-purpose chatbots. These findings provide evidence of the benefits of emerging solutions at the AI-optimization interface to expand the reach of optimization technologies in practice. They also uncover the impact of operations research expertise to facilitate interactive deployments through the design of effective and reliable optimization agents.