mlsanityData to 14 Oct 2026

Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models

arXiv 2406.12311 ↗First posted 18 Jun 2024cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Binarization, which converts weight parameters to binary values, has emerged as an effective strategy to reduce the size of large language models (LLMs). However, typical binarization techniques significantly diminish linguistic effectiveness of LLMs. To address this issue, we introduce a novel binarization technique called Mixture of Scales (BinaryMoS). Unlike conventional methods, BinaryMoS employs multiple scaling experts for binary weights, dynamically merging these experts for each token to adaptively generate scaling factors. This token-adaptive approach boosts the representational power of binarized LLMs by enabling contextual adjustments to the values of binary weights. Moreover, because this adaptive process only involves the scaling factors rather than the entire weight matrix, BinaryMoS maintains compression efficiency similar to traditional static binarization methods. Our experimental results reveal that BinaryMoS surpasses conventional binarization techniques in various natural language processing tasks and even outperforms 2-bit quantization methods, all while maintaining similar model size to static binarization techniques.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

32 of 37 references.

References · 32

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
20 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  7. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  8. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  9. 2023A Survey on Model Compression for Large Language Modelscited by 269
  10. 2020Binary Neural Networks: A Surveycited by 190
  11. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  12. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  13. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  14. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  15. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  16. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  17. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  18. 2024Evaluating Quantized Large Language Modelscited by 77
  19. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  20. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inferencecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  2. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  3. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  4. 2024CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMscited by 7
  5. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  6. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 4
  7. 2025HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMscited by 4
  8. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  9. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  10. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMscited by 2
  11. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Modelscited by 1
  12. 2026NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Modelscited by 1
2 more
  1. 2025Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddingscited by 0
  2. 2026RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
  2. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMs
  3. 2025Addition is almost all you need: Compressing large language models with double binary factorization
  4. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
  5. 2025Binary Neural Networks for Large Language Model: A Survey
  6. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
  7. 2024Data-free Weight Compress and Denoise for Large Language Models
  8. 2024CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs