# Evaluating Kubernetes Performance for GenAI Inference: From Automatic Speech Recognition to LLM Summarization

arXiv: https://arxiv.org/abs/2602.04900 · HTML: https://mlsanity.com/paper/2602.04900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 56 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 8 of 8)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [KIS-S: A GPU-Aware Kubernetes Inference Simulator with RL-Based Auto-Scaling](https://mlsanity.com/paper/2507.07932/) (2025)
- [MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters](https://mlsanity.com/paper/2303.13803/) (2023)
- [On the Cost of Model-Serving Frameworks: An Experimental Evaluation](https://mlsanity.com/paper/2411.10337/) (2024)
- [Cloud Native System for LLM Inference Serving](https://mlsanity.com/paper/2507.18007/) (2025)

## Abstract (arXiv)

As Generative AI (GenAI), particularly inference, rapidly emerges as a dominant workload category, the Kubernetes ecosystem is proactively evolving to natively support its unique demands. This industry paper demonstrates how emerging Kubernetes-native projects can be combined to deliver the benefits of container orchestration, such as scalability and resource efficiency, to complex AI workflows. We implement and evaluate an illustrative, multi-stage use case consisting of automatic speech recognition and summarization. First, we address batch inference by using Kueue to manage jobs that transcribe audio files with Whisper models and Dynamic Accelerator Slicer (DAS) to increase parallel job execution. Second, we address a discrete online inference scenario by feeding the transcripts to a Large Language Model for summarization hosted using llm-d, a novel solution utilizing the recent developments around the Kubernetes Gateway API Inference Extension (GAIE) for optimized routing of inference requests. Our findings illustrate that these complementary components (Kueue, DAS, and GAIE) form a cohesive, high-performance platform, proving Kubernetes' capability to serve as a unified foundation for demanding GenAI workloads: Kueue reduced total makespan by up to 15%; DAS shortened mean job completion time by 36%; and GAIE working in conjunction with llm-d improved tail Time to First Token latency by up to 90% even under high loads.