# DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs

arXiv: https://arxiv.org/abs/2312.05215 · HTML: https://mlsanity.com/paper/2312.05215/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 82 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 29 of 29)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks](https://mlsanity.com/paper/2102.08124/) (2021)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Punica: Multi-Tenant LoRA Serving](https://mlsanity.com/paper/2310.18547/) (2023)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study](https://mlsanity.com/paper/2307.08072/) (2023)
- [ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation](https://mlsanity.com/paper/1910.12129/) (2019)
- [INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation](https://mlsanity.com/paper/2306.08162/) (2023)

## Built on by (most cited first, 18 of 18)

- [BitDelta: Your Fine-Tune May Only Be Worth One Bit](https://mlsanity.com/paper/2402.10193/) (2024)
- [RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs](https://mlsanity.com/paper/2510.19225/) (2025)
- [TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling](https://mlsanity.com/paper/2510.02758/) (2025)
- [ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs](https://mlsanity.com/paper/2504.13237/) (2025)
- [DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization](https://mlsanity.com/paper/2410.08666/) (2024)
- [ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models](https://mlsanity.com/paper/2406.09041/) (2024)
- [ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache](https://mlsanity.com/paper/2604.06370/) (2026)
- [SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving](https://mlsanity.com/paper/2603.02599/) (2026)
- [D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation](https://mlsanity.com/paper/2604.16940/) (2026)
- [SineLoRADelta: Sine-Activated Delta Compression](https://mlsanity.com/paper/2505.21895/) (2025)
- [Enhancing Delta Compression in LLMs via SVD-based Quantization Error Minimization](https://mlsanity.com/paper/2506.11087/) (2025)
- [LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflows](https://mlsanity.com/paper/2604.08123/) (2026)
- [SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrization](https://mlsanity.com/paper/2604.08368/) (2026)
- [TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models](https://mlsanity.com/paper/2604.17104/) (2026)
- [ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform](https://mlsanity.com/paper/2607.26566/) (2026)
- [Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems](https://mlsanity.com/paper/2608.24650/) (2026)
- [Block-Level Weight-Space Structure Persists Under Post-Training: An Empirical Study Across LLM Families](https://mlsanity.com/paper/2609.26147/) (2026)
- [JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs](https://mlsanity.com/paper/2610.19822/) (2026)

## Abstract (arXiv)

Fine-tuning large language models (LLMs) greatly improves model quality for downstream tasks. However, serving many fine-tuned LLMs concurrently is challenging due to the sporadic, bursty, and varying request patterns of different LLMs. To bridge this gap, we present DeltaZip, an LLM serving system that efficiently serves multiple full-parameter fine-tuned models concurrently by aggressively compressing model deltas by up to 10x while maintaining high model quality. The key insight behind this design is that fine-tuning results in small-magnitude changes to the pre-trained model. By co-designing the serving system with the compression algorithm, DeltaZip achieves 2x to 12x improvement in throughput compared to the state-of-the-art systems.