mlsanityData to 14 Oct 2026

BitDelta: Your Fine-Tune May Only Be Worth One Bit

arXiv 2402.10193 ↗First posted 15 Feb 2024cs.LG cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks. Given the higher computational demand of pre-training, it's intuitive to assume that fine-tuning adds less new information to the model, and is thus more compressible. We explore this assumption by decomposing the weights of fine-tuned models into their pre-trained components and an additional delta. We introduce a simple method, BitDelta, which successfully quantizes this delta down to 1 bit without compromising performance. This interesting finding not only highlights the potential redundancy of information added during fine-tuning, but also has significant implications for the multi-tenant serving and multi-tenant storage of fine-tuned models. By enabling the use of a single high-precision base model accompanied by multiple 1-bit deltas, BitDelta dramatically reduces GPU memory requirements by more than 10x, which can also be translated to enhanced generation latency in multi-tenant settings. We validate BitDelta through experiments across Llama-2 and Mistral model families, and on models up to 70B parameters, showcasing minimal performance degradation over all tested settings.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

40 of 62 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
28 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  14. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  15. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  16. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  17. 2017Learning multiple visual domains with residual adapterscited by 557
  18. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  19. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  20. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  21. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  22. 2021Accelerating Sparse Deep Neural Networkscited by 223
  23. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  24. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  25. 2023Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Servicescited by 76
  26. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  27. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 18
  28. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 13; 2026: 11.

39 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  3. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  4. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  5. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  6. 2024Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Modelscited by 17
  7. 2025Personality Vector: Modulating Personality of Large Language Models by Model Mergingcited by 17
  8. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 13
  9. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 13
  10. 2024Quantized Delta Weight Is Safety Keepercited by 12
  11. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10
  12. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
27 more
  1. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusioncited by 5
  2. 20251bit-Merging: Dynamic Quantized Merging for Large Language Modelscited by 5
  3. 2025ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMscited by 5
  4. 2026Operationalising the Superficial Alignment Hypothesis via Task Complexitycited by 4
  5. 2024Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptationcited by 3
  6. 2024DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantizationcited by 3
  7. 2024ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Modelscited by 2
  8. 2024Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuningcited by 2
  9. 2024A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Modelscited by 2
  10. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  11. 2026PoLoRA: A Preconditioned Orthogonalized LoRA Optimizercited by 2
  12. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  13. 2025Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transformcited by 1
  14. 2025Dynamic Base model Shift for Delta Compressioncited by 1
  15. 2026D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximationcited by 1
  16. 2026Dynamic Model Merging Made Slimcited by 1
  17. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  18. 2026In-Cell Learning: Language Models That Update Their Own Weights in Sequence Without Changing the File They Shipcited by 1
  19. 2025RanDeS: Randomized Delta Superposition for Multi-Model Compressioncited by 0
  20. 2025SineLoRADelta: Sine-Activated Delta Compressioncited by 0
  21. 2025Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimescited by 0
  22. 2025Per-Axis Weight Deltas for Frequent Model Updatescited by 0
  23. 2026SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrizationcited by 0
  24. 2026BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deploymentcited by 0
  25. 2026Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compressioncited by 0
  26. 2026Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuningcited by 0
  27. 2026AI as a Compiler: Compiling Triton kernels without the Triton compilercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  2. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  3. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale
  4. 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
  5. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Model
  6. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  7. 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
  8. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits