mlsanityData to 7 Oct 2026

iServe: An Intent-based Serving System for LLMs

arXiv 2501.13111 ↗First posted 8 Jan 2025cs.SE cs.LGcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models (LLMs) are becoming ubiquitous across industries, where applications demand they fulfill diverse user intents. However, developers currently face the challenge of manually exploring numerous deployment configurations - combinations of parallelism and compression techniques that impact resource usage, latency, cost, and accuracy - to meet these intents. Assessing the impact of these configurations on user metrics requires extensive, costly profiling for each model. Existing approaches avoid this expense by using fixed, static configurations, but this often leads to sub-optimal performance and higher costs. Moreover, none of these solutions dynamically adapt to changing user intents to balance latency and cost, effectively. We present iServe, an automated, intent-based system for distributed LLM inference. Instead of manually selecting deployment configurations, developers simply specify their intent - such as minimizing latency, reducing cost, or meeting specific targets for either. iServe introduces fingerprints, lightweight representations of LLMs, to efficiently estimate how different configurations impact latency and memory usage. Based on these insights and GPU availability, iServe dynamically selects the optimal configuration to align with the user's intent. For various LLMs and query arrival rates, iServe best meets user intents compared to state-of-the-art systems by reducing latency by 77.62% and SLO violations by 7.09x while improving GPU throughput by 4.72x. Moreover, iServe's fingerprint-based profiling reduces profiling cost by 6.05x (GPU-hours) compared to baselines.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 66 of its 124 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

66 of 124 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems9 Jul
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023Fairness in Serving Large Language Models31 Dec
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  8. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 58

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2024Mixtral of Expertscited by 1,785
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
38 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2023The Falcon Series of Open Language Modelscited by 526
  7. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  10. 2022UL2: Unifying Language Learning Paradigmscited by 292
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  12. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  13. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  14. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  15. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  16. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  17. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  18. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  19. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  20. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  21. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  22. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  23. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  24. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  25. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  26. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  27. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  28. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51
  29. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  30. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  31. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
  32. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  33. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  34. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  35. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  36. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  37. 2021Tesseract: Parallelize the Tensor Parallelism Efficientlycited by 19
  38. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 19

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  2. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  3. 2026FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
  4. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  5. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
  6. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  7. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving