mlsanityData to 7 Oct 2026

BitNet a4.8: 4-bit Activations for 1-bit LLMs

arXiv 2411.04965 ↗First posted 7 Nov 2024cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 12 of its 22 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

12 of 22 references.

References · 12

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  5. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  6. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  7. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  8. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  9. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  10. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  11. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  12. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  2. 2025BitNet b1.58 2B4T Technical Reportcited by 30
  3. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  4. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 17
  5. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 16
  6. 2025Scaling Law for Quantization-Aware Trainingcited by 15
  7. 2026Attn-QAT: 4-Bit Attention With Quantization-Aware Trainingcited by 5
  8. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  9. 2025DBellQuant: Breaking the Bell with Double-Bell Transformation for LLMs Post Training Binarizationcited by 4
  10. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  11. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  12. 2025BitROM: Weight Reload-Free CiROM Architecture Towards Billion-Parameter 1.58-bit LLM Inferencecited by 1
4 more
  1. 2025Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devicescited by 1
  2. 2025ML For Hardware Design Interpretability: Challenges and Opportunitiescited by 0
  3. 2025TAT-VPR: Ternary Adaptive Transformer for Dynamic and Efficient Visual Place Recognitioncited by 0
  4. 2026FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Foldingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
  2. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
  3. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
  4. 2023SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
  5. 2025BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
  6. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?
  7. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  8. 2023Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models