mlsanityData to 14 Oct 2026

Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving

arXiv 2604.15732 ↗First posted 17 Apr 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Distributed LLM serving systems optimize per-request latency and throughput. However, under long-context workloads, inference accuracy becomes more variable. When incorrect responses trigger retries, accuracy directly translates into cumulative user-visible delay that is not captured by single-shot latency metrics. In this work, we argue that under long-context serving, accuracy becomes speed through retry dynamics. We introduce textitTime-to-Correct-Answer (TTCA), a metric that measures the wall-clock time required to obtain the first correct response. Our measurement study shows that prompt characteristics such as length and language amplify accuracy variance, which inflates TTCA. We demonstrate textitLightweight Accuracy-Aware Routing (LAAR), a capability-based routing design that reduces TTCA. Our results suggest that in long-context distributed serving, accuracy should be treated as a first-class systems objective.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 26 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

17 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving6 Feb

Other references · 16

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  6. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  7. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  8. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  9. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  10. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  11. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  12. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
4 more
  1. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  2. 2025When to Reason: Semantic Router for vLLMcited by 12
  3. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inferencecited by 10
  4. 2025Context Engineering for Multi-Agent LLM Code Assistants Using Elicit, NotebookLM, ChatGPT, and Claude Codecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  2. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
  3. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  4. 2025Niyama : Breaking the Silos of LLM Inference Serving
  5. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving
  6. 2025Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
  7. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
  8. 2024Performance Characterization of Expert Router for Scalable LLM Inference