# DB-LLM: Accurate Dual-Binarization for Efficient LLMs

arXiv: https://arxiv.org/abs/2402.11960 · HTML: https://mlsanity.com/paper/2402.11960/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2024.

## Built on by (most cited first, 30 of 37)

- [EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://mlsanity.com/paper/2407.11062/) (2024)
- [An empirical study of LLaMA3 quantization: from LLMs to MLLMs](https://mlsanity.com/paper/2404.14047/) (2024)
- [SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2405.14917/) (2024)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [PTQ4SAM: Post-Training Quantization for Segment Anything](https://mlsanity.com/paper/2405.03144/) (2024)
- [STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs](https://mlsanity.com/paper/2408.01803/) (2024)
- [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms](https://mlsanity.com/paper/2409.16694/) (2024)
- [Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs](https://mlsanity.com/paper/2508.14896/) (2025)
- [TernaryLLM: Ternarized Large Language Model](https://mlsanity.com/paper/2406.07177/) (2024)
- [QVD: Post-training Quantization for Video Diffusion Models](https://mlsanity.com/paper/2407.11585/) (2024)
- [Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models](https://mlsanity.com/paper/2510.03274/) (2025)
- [BinaryDM: Accurate Weight Binarization for Efficient Diffusion Models](https://mlsanity.com/paper/2404.05662/) (2024)
- [SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models](https://mlsanity.com/paper/2508.03351/) (2025)
- [PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling](https://mlsanity.com/paper/2506.05432/) (2025)
- [LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit](https://mlsanity.com/paper/2508.09981/) (2025)
- [Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets](https://mlsanity.com/paper/2603.16731/) (2026)
- [GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling](https://mlsanity.com/paper/2604.18556/) (2026)
- [CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs](https://mlsanity.com/paper/2606.26650/) (2026)
- [Progressive Binarization with Semi-Structured Pruning for LLMs](https://mlsanity.com/paper/2502.01705/) (2025)
- [Event-Priori-Based Vision-Language Model for Efficient Visual Understanding](https://mlsanity.com/paper/2506.07627/) (2025)
- [Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash](https://mlsanity.com/paper/2504.08378/) (2025)
- [HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs](https://mlsanity.com/paper/2512.00862/) (2025)
- [Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining](https://mlsanity.com/paper/2504.13932/) (2025)
- [Highly Efficient and Effective LLMs with Multi-Boolean Architectures](https://mlsanity.com/paper/2505.22811/) (2025)
- [RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models](https://mlsanity.com/paper/2510.01240/) (2025)
- [ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning](https://mlsanity.com/paper/2503.13089/) (2025)
- [Resource-Efficient Language Models: Quantization for Fast and Accessible Inference](https://mlsanity.com/paper/2505.08620/) (2025)
- [Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models](https://mlsanity.com/paper/2508.06974/) (2025)
- [LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss](https://mlsanity.com/paper/2605.08755/) (2026)
- [Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression](https://mlsanity.com/paper/2510.18650/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have significantly advanced the field of natural language processing, while the expensive memory and computation consumption impede their practical deployment. Quantization emerges as one of the most effective methods for improving the computational efficiency of LLMs. However, existing ultra-low-bit quantization always causes severe accuracy drops. In this paper, we empirically relieve the micro and macro characteristics of ultra-low bit quantization and present a novel Dual-Binarization method for LLMs, namely DB-LLM. For the micro-level, we take both the accuracy advantage of 2-bit-width and the efficiency advantage of binarization into account, introducing Flexible Dual Binarization (FDB). By splitting 2-bit quantized weights into two independent sets of binaries, FDB ensures the accuracy of representations and introduces flexibility, utilizing the efficient bitwise operations of binarization while retaining the inherent high sparsity of ultra-low bit quantization. For the macro-level, we find the distortion that exists in the prediction of LLM after quantization, which is specified as the deviations related to the ambiguity of samples. We propose the Deviation-Aware Distillation (DAD) method, enabling the model to focus differently on various samples. Comprehensive experiments show that our DB-LLM not only significantly surpasses the current State-of-The-Art (SoTA) in ultra-low bit quantization (eg, perplexity decreased from 9.64 to 7.23), but also achieves an additional 20% reduction in computational consumption compared to the SOTA method under the same bit-width. Our code will be released soon.