GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 28 references that are in the map. First posted on arXiv 12 Feb 2026.
Builds on
16 of 28 references.
Closest ancestors · 5
Same direction, newest first
- 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure15 Oct
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
- 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
- 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
Other references · 11
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
- 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
- 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
- 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
- 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 19
- 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 9
- 2025LLM Query Scheduling with Prefix Reuse and Latency Constraintscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
- 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
- 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
- 2026CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
- 2026WANSpec: Leveraging Global Compute Capacity for LLM Inference
- 2026Towards Distributed Inference of LLMs on a P2P Network
- 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
- 2026Cross-Model Autoscaling for Shared LLM Serving