mlsanityData to 7 Oct 2026

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

arXiv 2603.02599 ↗First posted 3 Mar 2026cs.AI cs.LGcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 38 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

26 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
12 more
  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  4. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  5. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  6. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  7. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  8. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  9. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 86
  10. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  11. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
  12. 2024Hermes: A Large Language Model Framework on the Journey to Autonomous Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Disaggregated Quantization: Specializing LLM Prefill and Decodecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
  3. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  4. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  5. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  6. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  7. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
  8. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving