mlsanityData to 14 Oct 2026

Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

arXiv 2604.02666 ↗First posted 3 Apr 2026cs.AI math.OCcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations. We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker. We generate thousands of conversations in a school scheduling case study. Results show that one-shot evaluation is severely limiting: the same optimization agent converges to much higher-quality solutions through conversations. Then, this paper uses this methodology to demonstrate that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions, as compared to general-purpose chatbots. These findings provide evidence of the benefits of emerging solutions at the AI-optimization interface to expand the reach of optimization technologies in practice. They also uncover the impact of operations research expertise to facilitate interactive deployments through the design of effective and reliable optimization agents.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 20 of its 41 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

20 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Large-Scale Optimization Model Auto-Formulation: Harnessing LLM Flexibility via Structured Workflow14 Jan

Other references · 19

  1. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  2. 2023Role-Play with Large Language Modelscited by 294
  3. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 135
  4. 2024ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modelingcited by 124
  5. 2024LLMOPT: Learning to Define and Solve General Optimization Problems from Scratchcited by 102
  6. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  7. 2024OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modelingcited by 68
  8. 2022Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptionscited by 54
  9. 2024Autoformulation of Mathematical Optimization Models Using LLMscited by 48
  10. 2025Large Language Models Pass the Turing Testcited by 48
  11. 2023OptiMUS: Optimization Modeling Using MIP Solvers and large language modelscited by 39
  12. 2024OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scalecited by 38
7 more
  1. 2022JuMP 1.0: Recent improvements to a modeling language for mathematical optimizationcited by 37
  2. 2023"I Want It That Way": Enabling Interactive Decision Support Using Large Language Models and Constraint Programmingcited by 19
  3. 2025AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Librarycited by 14
  4. 2023A Better Match for Drivers and Riders: Reinforcement Learning at Lyftcited by 9
  5. 2025Decision Information Meets Large Language Models: The Future of Explainable Operations Researchcited by 9
  6. 2025ORThought: Benchmarking and Automating Logistics Optimization Modelingcited by 5
  7. 2025Online Rack Placement in Large-Scale Data Centers: Online Sampling Optimization and Deploymentcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Democratizing Large-Scale Re-Optimization with LLM-Guided Model Patchescited by 2
  2. 2026FLARE: Verifying MILP Reformulations with LLM-Based Theorem Provingcited by 1
  3. 2026Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents
  2. 2026From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling
  3. 2026Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
  4. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  5. 2025OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM
  6. 2025Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development
  7. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
  8. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights