mlsanityData to 14 Oct 2026

PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving

arXiv 2609.10372 ↗First posted 9 Sep 2026cs.CV cs.AI cs.ROcited by 0

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

We present the PACE, a framework for retrieval-augmented dialogue serving that formalizes Perceived Time-to-First-Response (PTFR) as a QoE objective and minimizes it under quality/cost constraints. Unlike prior work on cascaded routing, semantic caching, or adaptive retrieval, PACE jointly controls which answer source composes the response and what fills the waiting window. Deployed on a humanoid-robot sales service, it combines three mechanisms: a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission. On 75k CarQA requests, the cascade halves pure-LLM PTFR at P95 (0.29 vs 0.53s at c16). The adaptive controller reaches 0.41s P95, outperforming RAG by 2.4 times at high load with equal quality. The filler controller cuts calls by 94% with zero conflict. Volatility-aware admission reduces stale answers from 86% to 0%. A gating rule ensures the controller never worse than the baseline, with exposure bounded by one hold period. This is the first quantification of filler-answer conflict risk in deployed services.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 26 of its 44 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

26 of 44 references.

References · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  11. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 702
14 more
  1. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  3. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  4. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  6. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  7. 2024Corrective Retrieval Augmented Generationcited by 197
  8. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  9. 2023AutoMix: Automatically Mixing Language Modelscited by 125
  10. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  11. 2025vCache: Verified Semantic Prompt Cachingcited by 17
  12. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  13. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agentscited by 2
  14. 2026Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation
  2. 2025TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
  3. 2026MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents
  4. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
  5. 2026Chronos: Temporal-Aware Conversational Agents with Structured Event Retrieval for Long-Term Memory
  6. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
  7. 2026When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents
  8. 2026Inference-Time Budget Control for LLM Search Agents