mlsanityData to 14 Oct 2026

NanoFlow: Towards Optimal Large Language Model Serving Throughput

arXiv 2408.12757 ↗First posted 22 Aug 2024cs.DCcited by 90

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) have resulted in a surging demand for planet-scale serving systems, where tens of thousands of GPUs continuously serve hundreds of millions of users. Consequently, throughput has emerged as a key metric that determines serving systems' performance. Due to large model sizes and memory-intensive self-attention, LLM serving has been commonly assumed to be memory-bound. Through a detailed analysis, we show that despite having memory-intensive components, end-to-end LLM serving is compute bound for most common workloads and LLMs. Alas, most existing serving engines fall short from optimal compute utilization, because the heterogeneous operations that comprise LLM serving--compute, memory, networking--are executed sequentially within a device. We propose NanoFlow, a novel serving framework that exploits intra-device parallelism, which overlaps the usage of heterogeneous resources within a single device. NanoFlow splits inputs into smaller nano-batches and duplicates operations to operate on each portion independently, enabling overlapping. NanoFlow automatically identifies the number, size, ordering, and GPU resource allocation of nano-batches to minimize the execution time, while considering the interference of concurrent operations. We evaluate NanoFlow's end-to-end serving throughput on several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc. With practical workloads, NanoFlow provides 1.91x throughput boost compared to state-of-the-art serving systems achieving 50% to 72% of optimal throughput across popular models.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 63 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

22 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2016Layer Normalizationcited by 5,669
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
7 more
  1. 2022Efficiently Scaling Transformer Inferencecited by 539
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  5. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  6. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  7. 2020Automatic Horizontal Fusion for GPU Kernelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 39; 2026: 43.

90 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2025LongCat-Flash Technical Reportcited by 58
  3. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  4. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  5. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  6. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  7. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  8. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  9. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMscited by 24
  10. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  11. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  12. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
38 more
  1. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  2. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  3. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 16
  4. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  5. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
  6. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 15
  7. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 13
  8. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12
  9. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
  10. 2025Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reorderingcited by 10
  11. 2025On Evaluating Performance of LLM Inference Serving Systemscited by 10
  12. 2025Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decodingcited by 10
  13. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  14. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6
  15. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inferencecited by 6
  16. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 6
  17. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 6
  18. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  19. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  20. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  21. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Servingcited by 5
  22. 2025Compass: SLO-aware Query Planner for Compound AI Serving at Scalecited by 5
  23. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokenscited by 5
  24. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  25. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  26. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Servingcited by 4
  27. 2025LEANN: A Low-Storage Vector Indexcited by 4
  28. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  29. 2025Dense Video Understanding with Gated Residual Tokenizationcited by 4
  30. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
  31. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  32. 2026Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernelcited by 4
  33. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  34. 2025xLLM Technical Reportcited by 3
  35. 2026Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Trainingcited by 3
  36. 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
  37. 2026Bole: Efficient Tree Speculation for Hybrid-Attention Language Modelscited by 3
  38. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  2. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  4. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  5. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  6. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving
  7. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  8. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration