# Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing

arXiv: https://arxiv.org/abs/2408.13510 · HTML: https://mlsanity.com/paper/2408.13510/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2024.

## Built on by (most cited first, 7 of 7)

- [Optimal Scheduling Algorithms for LLM Inference: Theory and Practice](https://mlsanity.com/paper/2508.01002/) (2025)
- [LAPS: A Length-Aware-Prefill LLM Serving System](https://mlsanity.com/paper/2601.11589/) (2026)
- [MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference](https://mlsanity.com/paper/2504.09775/) (2025)
- [Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Framework](https://mlsanity.com/paper/2509.26534/) (2025)
- [Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving](https://mlsanity.com/paper/2606.01839/) (2026)
- [BeLLMan: Controlling LLM Congestion](https://mlsanity.com/paper/2510.15330/) (2025)
- [TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference](https://mlsanity.com/paper/2603.26498/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) workloads have distinct prefill and decode phases with different compute and memory requirements which should ideally be accounted for when scheduling input queries across different LLM instances in a cluster. However existing scheduling algorithms treat LLM workloads as monolithic jobs without considering the distinct characteristics of the two phases in each workload. This leads to sub-optimal scheduling and increased response latency. In this work, we start by characterizing factors affecting the response latency during LLM inference serving. We establish that better load balancing of inference requests across the available LLM instances can improve the end-to-end latency to a larger extent than merely focusing on optimizing the instance-level scheduler. Motivated by our findings, we propose a heuristic-guided reinforcement learning-based intelligent router for data-driven and workload-aware scheduling. Our router schedules queries across LLM instances by leveraging a trainable response-length predictor, and a novel formulation for estimating the impact of mixing different workloads and achieves over 11% lower end-to-end latency than existing approaches on a mix of public datasets and 7.8% lower end-to-end latency on real workload data with diverse input and output trends from Cloud Provider X. Additionally, the proposed framework can also serve as a standard for benchmarking different LLM inference schedulers since it provides the best latency for a given model, hardware, and instance-level scheduler combination.