mlsanityData to 7 Oct 2026

OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents

arXiv 2605.28158 ↗First posted 27 May 2026cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained problem statement into a mathematical formulation or solver program. Such settings abstract away two characteristics of real industrial OR workflows: persistent multi-artifact workspaces and multi-stage task lifecycles. We introduce OR-Space, a full-lifecycle workspace benchmark for evaluating industrial optimization agents across model construction, model revision, and grounded explanation. Each instance is an executable workspace containing business documents, structured data, optional code artifacts, solver outputs, and task-specific evaluators distributed across interdependent files. OR-Space defines three task modes: Build, where agents construct solver-ready optimization models from heterogeneous artifacts; Revise, where agents modify existing models under changing requirements or solver feedback while preserving valid prior logic; and Explain, where agents answer grounded questions about solutions, constraints, and business implications using evidence spread across workspace artifacts. By combining persistent workspaces with lifecycle-oriented tasks, OR-Space evaluates whether agents can perform reliable optimization work beyond end-to-end text generation. We describe the benchmark design, evaluation protocol, and quality-control pipeline, and position OR-Space as a benchmark for studying the reliability, failure modes, and practical readiness of LLM agents in industrial OR workflows.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 33 of its 45 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

33 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling28 Apr
  2. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey6 Jun
  3. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROI23 May
  4. 2025OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM13 Mar
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  7. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2018Datasheets for Datasetscited by 1,371
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  11. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  12. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
14 more
  1. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  2. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  3. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  4. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 133
  5. 2024ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modelingcited by 124
  6. 2024OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Modelscited by 105
  7. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  8. 2015JuMP: A Modeling Language for Mathematical Optimizationcited by 49
  9. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 46
  10. 2025StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Modelscited by 26
  11. 2025Auto-Formulating Dynamic Programming Problems with Large Language Modelscited by 15
  12. 2024Evaluating LLM Reasoning in the Operations Research Domain with ORQAcited by 12
  13. 2026Constructing Industrial-Scale Optimization Modeling Benchmarkcited by 10
  14. 2026InvEvolve: Evolving White-Box Inventory Policies via Large Language Models with Performance Guaranteescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
  2. 2026ORPilot: A Production-Oriented Agentic LLM-for-OR Tool for Optimization Modeling
  3. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents
  4. 2026ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
  5. 2026AgentFactory: Towards Automated Agentic System Design and Optimization
  6. 2026MIRROR: A Multi-Agent Framework with Iterative Adaptive Revision and Hierarchical Retrieval for Optimization Modeling in Operations Research
  7. 2026Workspace Optimization: How to Train Your Agent
  8. 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflows