mlsanityData to 7 Oct 2026

Seesaw: High-throughput LLM Inference via Model Re-sharding

arXiv 2503.06433 ↗First posted 9 Mar 2025cs.DC cs.AIcited by 15

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed. However, the distinct computational characteristics inherent in the two stages of LLM inference-prefilling and decoding-render a single static parallelization strategy insufficient for the effective optimization of both stages. In this work, we present Seesaw, an LLM inference engine optimized for throughput-oriented tasks. The key idea behind Seesaw is dynamic model re-sharding, a technique that facilitates the dynamic reconfiguration of parallelization strategies across stages, thereby maximizing throughput at both phases. To mitigate re-sharding overhead and optimize computational efficiency, we employ tiered KV cache buffering and transition-minimizing scheduling. These approaches work synergistically to reduce the overhead caused by frequent stage transitions while ensuring maximum batching efficiency. Our evaluation demonstrates that Seesaw achieves a throughput increase of up to 1.78x (1.36x on average) compared to vLLM, the most widely used state-of-the-art LLM inference engine.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 78 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

44 of 78 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024A System for Microserving of LLMs17 Dec
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  3. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  4. 2024Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving19 Jun
  5. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines18 Mar
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  7. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization2 Mar
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  11. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec
  12. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
1 more
  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
19 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  4. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  5. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  7. 2022Can Foundation Models Wrangle Your Data?cited by 173
  8. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  9. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  10. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  11. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  12. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  13. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  14. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  15. 2024FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusioncited by 53
  16. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  17. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  18. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  19. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  2. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  3. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  4. 2025Military AI Cyber Agents (MAICAs) Constitute a Global Threat to Critical Infrastructurecited by 2
  5. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  6. 202510Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Trainingcited by 2
  7. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  8. 2026Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurationscited by 1
  9. 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inferencecited by 0
  10. 2025DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architecturescited by 0
  11. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Serverscited by 0
  12. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
3 more
  1. 2026KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMscited by 0
  2. 2026Efficient Vision-Language-Action Management and Serving for Robot Factoriescited by 0
  3. 2026SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Glinthawk: A Two-Tiered Architecture for Offline LLM Inference
  2. 2024Multi-Bin Batching for Increasing LLM Inference Throughput
  3. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  4. 2023Efficient LLM Inference on CPUs
  5. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  8. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference