mlsanityData to 7 Oct 2026

OneBit: Towards Extremely Low-bit Large Language Models

arXiv 2402.11295 ↗First posted 17 Feb 2024cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

Model quantification uses low bit-width values to represent the weight matrices of existing models to be quantized, which is a promising approach to reduce both storage and computational overheads of deploying highly anticipated LLMs. However, current quantization methods suffer severe performance degradation when the bit-width is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to quantize models. This paper boldly quantizes the weight matrices of LLMs to 1-bit, paving the way for the extremely low bit-width deployment of LLMs. For this target, we introduce a 1-bit model compressing framework named OneBit, including a novel 1-bit parameter representation method to better quantize LLMs as well as an effective parameter initialization method based on matrix decomposition to improve the convergence speed of the quantization framework. Sufficient experimental results indicate that OneBit achieves good performance (at least 81% of the non-quantized performance on LLaMA models) with robust training processes when only using 1-bit weight matrices.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
24 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  10. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  13. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  14. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  15. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  16. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  17. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  18. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  19. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  20. 2023A Survey on Model Compression for Large Language Modelscited by 263
  21. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  22. 2023Efficient Large Language Models: A Surveycited by 183
  23. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  24. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 30; 2026: 28.

77 papers.

Sort:

By area:

  1. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  2. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  3. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  4. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  5. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  6. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  7. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  8. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  9. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  10. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  11. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 30
  12. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
38 more
  1. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  2. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14
  3. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Servingcited by 13
  4. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12
  5. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  6. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  7. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Modelscited by 10
  8. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  9. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  10. 2024CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMscited by 7
  11. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  12. 2024Direct Quantized Training of Language Models with Stochastic Roundingcited by 5
  13. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  14. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  15. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 5
  16. 2026SliderQuant: Accurate Post-Training Quantization for LLMscited by 5
  17. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMscited by 5
  18. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4
  19. 2025T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganizationcited by 4
  20. 2025HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMscited by 4
  21. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  22. 2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
  23. 2026OneComp: One-Line Revolution for Generative AI Model Compressioncited by 3
  24. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  25. 2025DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weightscited by 2
  26. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  27. 2025Boost Post-Training Quantization via Null Space Optimization for Large Language Modelscited by 2
  28. 2025Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Modelscited by 2
  29. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 2
  30. 2025Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigationcited by 2
  31. 2026Towards Understanding Best Practices for Quantization of Vision-Language Modelscited by 2
  32. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  33. 2024CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMscited by 1
  34. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  35. 2025PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMscited by 1
  36. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Modelscited by 1
  37. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Accelerationcited by 1
  38. 2026Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptationcited by 1

50 most cited of 77 All 77 (JSON)