# Serving Agentic Workflows with a Physical-Plan Compiler and Adaptive Runtime

arXiv: https://arxiv.org/abs/2607.02942 · HTML: https://mlsanity.com/paper/2607.02942/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Multi-agent Architecture Search via Agentic Supernet](https://mlsanity.com/paper/2502.04180/) (2025)
- [Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs](https://mlsanity.com/paper/2605.00674/) (2026)
- [ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario](https://mlsanity.com/paper/2501.10132/) (2025)
- [EvoAgentX: An Automated Framework for Evolving Agentic Workflows](https://mlsanity.com/paper/2507.03616/) (2025)
- [MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision](https://mlsanity.com/paper/2505.14996/) (2025)
- [Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms](https://mlsanity.com/paper/2508.18298/) (2025)
- [Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows](https://mlsanity.com/paper/2511.20975/) (2025)
- [Orla: A Library for Serving LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2603.13605/) (2026)

## Built on by (most cited first, 1 of 1)

- [JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs](https://mlsanity.com/paper/2610.03296/) (2026)

## Abstract (arXiv)

Efficient serving of agentic workflows requires selecting each LLM node's model, verification policy, and backend to balance output quality, latency, and throughput. These assignments must also adapt to changes in serving load. Existing approaches address parts of this problem through model routing, verifier placement, and backend scheduling. However, independent optimization overlooks their dependencies: model and backend choices determine verification cost, while verification changes the quality-cost trade-off among models. Ignoring these interactions can waste serving resources and degrade workflow performance. To address this problem, we propose Dyserve, which provides the missing workflow physical-planning layer between orchestration and model serving through compiler-runtime co-design. Our design is guided by three observations: planning headroom is request-dependent; node vulnerability, the impact of local errors on final correctness, depends on position and task type; and serving load changes the cost of a plan during execution. For each request, its profile-guided compiler jointly selects node implementations and prepares pressure-specialized variants for the materialized workflow. The runtime selects variants using live backend pressure and updates only undispatched assignments, without invoking the optimizer on the load-change path. Across four agentic workloads, Dyserve improves accuracy by 3-9 percentage points with 1.1-6.8× mean-latency speedups over the highest-accuracy evaluated baseline for each workload. On a burst trace, variant switching raises the fraction of correct, on-time completions from 18.1% to 67.2% relative to admission-only execution.