mlsanityData to 7 Oct 2026

SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips

arXiv 2601.20309 ↗First posted 28 Jan 2026cs.DC cs.AI cs.LGcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Model (LLM) serving faces a fundamental tension between stringent latency Service Level Objectives (SLOs) and limited GPU memory capacity. When high request rates exhaust the KV cache budget, existing LLM inference systems often suffer severe head-of-line (HOL) blocking. While prior work explored PCIe-based offloading, these approaches cannot sustain responsiveness under high request rates, often failing to meet tight Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT) SLOs. We present SuperInfer, a high-performance LLM inference system designed for emerging Superchips (e.g., NVIDIA GH200) with tightly coupled GPU-CPU architecture via NVLink-C2C. SuperInfer introduces RotaSched, the first proactive, SLO-aware rotary scheduler that rotates requests to maintain responsiveness on Superchips, and DuplexKV, an optimized rotation engine that enables full-duplex transfer over NVLink-C2C. Evaluations on GH200 using various models and datasets show that SuperInfer improves TTFT SLO attainment rates by up to 74.7% while maintaining comparable TBT and throughput compared to state-of-the-art systems, demonstrating that SLO-aware scheduling and memory co-design unlocks the full potential of Superchips for responsive LLM serving. Code is available in https://github.com/Supercomputing-System-AI-Lab/SuperInfer.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 63 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

40 of 63 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling3 Oct
  2. 2025Equinox: Holistic Fair Scheduling in Serving Large Language Models19 Aug
  3. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference2 Nov
  5. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  7. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  8. 2024Queue management for slo-oriented large language model serving5 Jun
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  10. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  12. 2023Fairness in Serving Large Language Models31 Dec
1 more
  1. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
15 more
  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  3. 2025Qwen2.5-1M Technical Reportcited by 158
  4. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  5. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16
  6. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
  7. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  8. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  9. 2024HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devicescited by 9
  10. 2024Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchipcited by 8
  11. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  12. 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
  13. 2025SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchipscited by 6
  14. 2025Memory Offloading for Large Language Model Inference with Latency SLO Guaranteescited by 3
  15. 2024Harnessing Integrated CPU-GPU System Memory for HPC: a first look into Grace Hoppercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 0
  2. 2026Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  2. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
  3. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  4. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  6. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
  7. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  8. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading