mlsanityData to 7 Oct 2026

Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services

arXiv 2404.16283 ↗First posted 25 Apr 2024cs.DC cs.LGcited by 44

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language models (LLMs) are now at the core of conversational AI services such as real-time translation and chatbots, which provide live user interaction by incrementally streaming text to the user. However, existing LLM serving systems fail to provide good user experience because their optimization metrics are not always aligned with user experience. In this paper, we first introduce and define the notion of Quality-of-Experience (QoE) for text streaming services by considering each user's end-to-end interaction timeline. Based on this, we propose Andes, a QoE-aware LLM serving system that enhances user experience by ensuring that users receive the first token promptly and subsequent tokens at a smooth, digestible pace, even during surge periods. This is enabled by Andes's preemptive request scheduler that dynamically prioritizes requests at the token granularity based on each request's expected QoE gain and GPU resource usage. Our evaluations demonstrate that, compared to state-of-the-art LLM serving systems, Andes improves the average QoE by up to 4.7× given the same GPU resource, or saves up to 61% GPU resources while maintaining the same high QoE.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 49 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

23 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  4. 2023Fairness in Serving Large Language Models31 Dec
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  11. 2023Efficient Large Language Models: A Surveycited by 183
  12. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
6 more
  1. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  2. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  3. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  4. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  5. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  6. 2023SuperServe: Fine-Grained Inference Serving for Unpredictable Workloadscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 23; 2026: 12.

44 papers.

Sort:

By area:

  1. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  2. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  3. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  4. 2024Queue management for slo-oriented large language model servingcited by 36
  5. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 31
  6. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Productioncited by 29
  7. 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
  8. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  9. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  10. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 14
  11. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  12. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
32 more
  1. 2025Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit Viewcited by 11
  2. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  3. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  4. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Servingcited by 5
  5. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Servingcited by 5
  6. 2025Compass: SLO-aware Query Planner for Compound AI Serving at Scalecited by 5
  7. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  8. 2025Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edgecited by 3
  9. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Servingcited by 3
  10. 2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
  11. 2025Streaming Looking Ahead with Token-level Self-rewardcited by 2
  12. 2025Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Servingcited by 2
  13. 2025ConsumerBench: Benchmarking Generative AI Applications on End-User Devicescited by 2
  14. 2026Sci-Reasoning: A Dataset Decoding AI Innovation Patternscited by 2
  15. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  16. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  17. 2024Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performancecited by 1
  18. 2025AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decodingcited by 1
  19. 2025Semantic Scheduling for LLM Inferencecited by 1
  20. 2026PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Modelscited by 1
  21. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  22. 2026EnerInfer: Energy-Aware On-Device LLM Inferencecited by 1
  23. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environmentscited by 0
  24. 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scalingcited by 0
  25. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  26. 2026Duration Aware Scheduling for ASR Serving Under Workload Driftcited by 0
  27. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demandscited by 0
  28. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  29. 2026Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexingcited by 0
  30. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inferencecited by 0
  31. 2026Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timingcited by 0
  32. 2026PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs
  2. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requests
  3. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents
  5. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  6. 2024Benchmarking Floworks against OpenAI & Anthropic: A Novel Framework for Enhanced LLM Function Calling
  7. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  8. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems