mlsanityData to 7 Oct 2026

Duration Aware Scheduling for ASR Serving Under Workload Drift

arXiv 2603.11273 ↗First posted 11 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Scheduling policies in large-scale Automatic Speech Recognition (ASR) serving pipelines play a key role in determining end-to-end (E2E) latency. Yet, widely used serving engines rely on first-come-first-served (FCFS) scheduling, which ignores variability in request duration and leads to head-of-line blocking under workload drift. We show that audio duration is an accurate proxy for job processing time in ASR models such as Whisper, and use this insight to enable duration-aware scheduling. We integrate two classical algorithms, Shortest Job First (SJF) and Highest Response Ratio Next (HRRN), into vLLM and evaluate them under realistic and drifted workloads. On LibriSpeech test-clean, compared to baseline, SJF reduces median E2E latency by up to 73% at high load, but increases 90th-percentile tail latency by up to 97% due to starvation of long requests. HRRN addresses this trade-off: it reduces median E2E latency by up to 28% while bounding tail-latency degradation to at most 24%. These gains persist under workload drift, with no throughput penalty and <0.1,ms scheduling overhead per request.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 25 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

15 of 25 references.

References · 15

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  7. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  8. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  9. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  10. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  11. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
  12. 2023Fairness in Serving Large Language Modelscited by 77
3 more
  1. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  2. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 44
  3. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20

Built on by

Not cited yet.