mlsanityData to 14 Oct 2026

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

arXiv 2504.18415 ↗First posted 25 Apr 2025cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

14 of 17 references.

References · 14

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  10. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  11. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  12. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
2 more
  1. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  2. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 43
  2. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  3. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 23
  4. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  5. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  6. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 10
  7. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantizationcited by 8
  8. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  9. 2025TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edgecited by 6
  10. 2026BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimizationcited by 6
  11. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMscited by 6
  12. 2025iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± icited by 5
6 more
  1. 2026Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantizationcited by 5
  2. 2026Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMscited by 3
  3. 2025A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Predictioncited by 2
  4. 2025Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± icited by 2
  5. 2025BitSkip: An Empirical Analysis of Quantization and Early Exit Composition in Transformerscited by 1
  6. 2026Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  2. 2026InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
  3. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
  4. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs
  5. 2025BitNet b1.58 2B4T Technical Report
  6. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference
  7. 2025SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
  8. 2024BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networks