# Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

arXiv: https://arxiv.org/abs/2608.12123 · HTML: https://mlsanity.com/paper/2608.12123/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 16 of 16)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms](https://mlsanity.com/paper/2508.18298/) (2025)
- [ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System](https://mlsanity.com/paper/2602.13692/) (2026)
- [MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs](https://mlsanity.com/paper/2512.22219/) (2025)
- [SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters](https://mlsanity.com/paper/2605.00528/) (2026)
- [SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms](https://mlsanity.com/paper/2301.12865/) (2023)
- [Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel](https://mlsanity.com/paper/2604.13327/) (2026)
- [TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving](https://mlsanity.com/paper/2607.29678/) (2026)
- [Joint Batching and Scheduling for High-Throughput Multiuser Edge AI with Asynchronous Task Arrivals](https://mlsanity.com/paper/2307.14350/) (2023)
- [Agentic CPU-GPU Scheduling for Heterogeneous AI Workloads](https://mlsanity.com/paper/2607.22242/) (2026)
- [Constraint Programming Models For Serial Batch Scheduling With Minimum Batch Size](https://mlsanity.com/paper/2504.08793/) (2025)
- [Architectural Implications of Agentic AI Workflows](https://mlsanity.com/paper/2608.04458/) (2026)
- [AgentServe: Algorithm-System Co-Design for Efficient Agentic AI Serving on a Consumer-Grade GPU](https://mlsanity.com/paper/2603.10342/) (2026)
- [GPUOS: A GPU Operating System Primitive for Transparent Operation Fusion](https://mlsanity.com/paper/2604.17861/) (2026)
- [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](https://mlsanity.com/paper/2608.08340/) (2026)

## Abstract (arXiv)

We introduce a mechanism that improves LLM-agent execution: keeping four-byte, GPU-computed control-path routing decisions on-device, avoiding a round-trip to host memory for redispatch. It is faster than host-dispatch in all 36 placements x mechanisms settings in our benchmark across four named GPUs with row-median speedup of 1.19x-2.39x, and outputs are correct, matching an independent host implementation, for all 14,557,440 calls in all configurations, in both tested mechanisms. Agent control paths can be deterministic. Their transitions between LLM calls and external tool calls can be GPU-accelerated by batching across agents and time. We define a cohort of transitions with launch times in a window, and study four shares of GPU-active time: a fixed-partition baseline F, an optimal P*, an upper bound U, and an online result A. Under assumptions of uniform relative launch deadlines, infinite capacity, and zero service times, we compute P* with a custom dynamic program and validate our simulator against it. We replay an anonymized 851-session trace, shown in full as a panel, stationary-ized with a Poisson process. With 100,000 target active agents and a launch deadline of 50 ms, where the headline results have K=256, we find F=30.19%, P*=43.00%, U=45.85%; our method for computing P* reveals that 81.83% of this opportunity is recoverable from cohort packing at fixed window ends. Our method for U relies on a novel notion of route-key, used only as a proxy (via conditional statistics) and does not purport to establish route-equivalence. We find a nested graph that retains some host-side decisions slower in all 60 configurations (five placements).