mlsanityData to 7 Oct 2026

BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models

arXiv 2404.18322 ↗First posted 28 Apr 2024cs.DCcited by 5

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The increasing demand for Large Language Models (LLMs) across various applications has led to a significant shift in the design of deep learning serving systems. Deploying LLMs, particularly in multi-tenant environments, poses substantial challenges due to their high computational and memory demands. We introduce BlockLLM, a serving system that leverages component sharing among fine-tuned LLM models to provide an efficient and flexible solution for LLM workloads. BlockLLM partitions models into finer-grained blocks, enabling the reuse of model components and independent provisioning to improve computation efficiency. BlockLLM comprises an offline block zoo for storing blocks and an online system to serve requests through chains of blocks. It offers multi-fold flexibilities: (1) Adaptive assembly of blocks on-the-fly through equivalence evaluation among blocks in the zoo; (2) Per-block batch size configuration and best-effort KV cache coordination at the individual block level; (3) Speculative execution and locality-aware block placement to reduce communication costs from dynamic block resource allocation. Our evaluation shows that BlockLLM reduces memory and storage footprints and improves computational efficiency, outperforming existing serving approach in 95%ile latency and GPU utilization by 33.5% and 20.1%, respectively, with minimal impact on accuracy

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 56 references that are in the map. First posted on arXiv 28 Apr 2024.

Builds on

33 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb
  2. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov

Other references · 31

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
19 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2022Efficiently Scaling Transformer Inferencecited by 529
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  12. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  13. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  14. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  15. 2023PaSS: Parallel Speculative Samplingcited by 49
  16. 2023Stitchable Neural Networkscited by 26
  17. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  18. 2018Mini-batch Serialization: CNN Training with Inter-layer Data Reusecited by 8
  19. 2023Cooperative Learning for Cost-Adaptive Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  2. 2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 8
  3. 2026Burn-After-Use for Preventing Data Leakage through a Secure Multi-Tenant Architecture in Enterprise LLMcited by 1
  4. 2025Efficient Large Language Models with Zero-Shot Adjustable Accelerationcited by 0
  5. 2026HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloadscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
  2. 2025Leveraging the true depth of LLMs
  3. 2026E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
  4. 2026BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
  5. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache
  6. 2024TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs
  7. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  8. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity