mlsanityData to 7 Oct 2026

Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism

arXiv 2605.05467 ↗First posted 6 May 2026cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM serving is increasingly multi-tenant: the same deployment must handle latency-critical interactive requests and more relaxed background workloads under a fixed GPU budget. This creates a tiered-SLO setting where maximizing overall goodput (requests that satisfy both TTFT and TPOT targets) is challenging because workload mix, request lengths, and load intensity vary over time. Existing systems mainly optimize request-level controls (e.g., queuing and batching) while keeping execution configuration largely static, which limits adaptation under multi-tier contention. We present Nitsum, a distributed LLM serving system that treats tensor parallelism (TP) as a first-class runtime control surface rather than a static deployment choice. Nitsum jointly optimizes TP level, prefill/decode GPU split, and request scheduling. To make frequent TP adaptation practical, Nitsum introduces TP-aware weight reuse and fast KV migration. Experiments on real traces and targeted microbenchmarks show that Nitsum improves SLO-compliant goodput over SoTA by up to 5.3 times.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 38 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

13 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  2. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines8 Aug
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  7. 2025Hierarchical Autoscaling for Large Language Model Serving with Chironcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 0
  2. 2026FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configurationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  2. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  3. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  4. 2026StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
  5. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
  6. 2026Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking ↗
  7. 2026NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
  8. 2026TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure