# EWSJF: An Adaptive Scheduler with Hybrid Partitioning for Mixed-Workload LLM Inference

arXiv: https://arxiv.org/abs/2601.21758 · HTML: https://mlsanity.com/paper/2601.21758/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 14 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 1 of 1)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)

## Built on by (most cited first, 2 of 2)

- [Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference](https://mlsanity.com/paper/2604.09613/) (2026)
- [Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving](https://mlsanity.com/paper/2604.08075/) (2026)

## Abstract (arXiv)

Serving Large Language Models (LLMs) under mixed workloads--short, latency-sensitive interactive queries alongside long, throughput-oriented batch requests--poses a fundamental scheduling challenge. Standard First-Come, First-Served (FCFS) policies suffer from severe head-of-line blocking, leading to high tail latency and underutilized hardware. We introduce EWSJF (Effective Workload-based Shortest Job First), an adaptive request-level scheduler that learns workload structure in real time to jointly improve fairness and throughput. EWSJF operates upstream of execution-level schedulers and integrates four components: (1) Refine-and-Prune, an unsupervised partitioning algorithm that discovers performance-homogeneous request groups; (2) Dynamic Queue Routing for assigning requests to these groups; (3) Density-Weighted Scoring, a context-aware prioritization function balancing urgency and fairness; and (4) Bayesian Meta-Optimization, which continuously tunes scoring and partitioning parameters based on live performance feedback. Implemented in vLLM, EWSJF improves end-to-end throughput by over 30% and reduces average Time-To-First-Token for short requests by up to 4x compared to FCFS. These results demonstrate that adaptive, learning-based request scheduling is a critical missing layer for efficient and responsive LLM serving. Implementation available at https://anonymous.4open.science/r/vllm_0110-32D8.