Substrate-Portable Execution for Production LLM Workflows
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 29 references that are in the map. First posted on arXiv 5 Sep 2026.
Builds on
10 of 29 references.
Closest ancestors · 1
Same direction, newest first
Other references · 9
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
- 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
- 2024PreAct: Prediction Enhances Agent's Planning Abilitycited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
- 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
- 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
- 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
- 2026Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
- 2026Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation
- 2026Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows
- 2026A Policy-Driven Runtime Layer for Agentic LLM Serving