# SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

arXiv: https://arxiv.org/abs/2603.02599 · HTML: https://mlsanity.com/paper/2603.02599/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers](https://mlsanity.com/paper/2402.19255/) (2024)
- [Llama-Nemotron: Efficient Reasoning Models](https://mlsanity.com/paper/2505.00949/) (2025)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains](https://mlsanity.com/paper/2501.05707/) (2025)
- [Small LLMs Are Weak Tool Learners: A Multi-LLM Agent](https://mlsanity.com/paper/2401.07324/) (2024)
- [AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2504.00587/) (2025)
- [ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration](https://mlsanity.com/paper/2511.21689/) (2025)
- [DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs](https://mlsanity.com/paper/2312.05215/) (2023)
- [Hermes: A Large Language Model Framework on the Journey to Autonomous Networks](https://mlsanity.com/paper/2411.06490/) (2024)

## Built on by (most cited first, 1 of 1)

- [Disaggregated Quantization: Specializing LLM Prefill and Decode](https://mlsanity.com/paper/2609.26333/) (2026)

## Abstract (arXiv)

In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.