# BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs

arXiv: https://arxiv.org/abs/2504.18415 · HTML: https://mlsanity.com/paper/2504.18415/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering](https://mlsanity.com/paper/1911.07176/) (2019)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://mlsanity.com/paper/2407.10969/) (2024)
- [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://mlsanity.com/paper/2411.04965/) (2024)

## Built on by (most cited first, 18 of 18)

- [BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation](https://mlsanity.com/paper/2506.07530/) (2025)
- [The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks](https://mlsanity.com/paper/2603.05498/) (2026)
- [Get Experience from Practice: LLM Agents with Record & Replay](https://mlsanity.com/paper/2505.17716/) (2025)
- [A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training](https://mlsanity.com/paper/2601.22966/) (2026)
- [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025)
- [What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study](https://mlsanity.com/paper/2601.14888/) (2026)
- [TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization](https://mlsanity.com/paper/2606.13054/) (2026)
- [NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache](https://mlsanity.com/paper/2505.18231/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization](https://mlsanity.com/paper/2603.16590/) (2026)
- [CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs](https://mlsanity.com/paper/2606.26650/) (2026)
- [iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± i](https://mlsanity.com/paper/2508.05571/) (2025)
- [Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization](https://mlsanity.com/paper/2608.01078/) (2026)
- [Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs](https://mlsanity.com/paper/2604.07888/) (2026)
- [A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Prediction](https://mlsanity.com/paper/2506.13678/) (2025)
- [Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± i](https://mlsanity.com/paper/2512.02901/) (2025)
- [BitSkip: An Empirical Analysis of Quantization and Early Exit Composition in Transformers](https://mlsanity.com/paper/2510.23766/) (2025)
- [Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference](https://mlsanity.com/paper/2604.25183/) (2026)

## Abstract (arXiv)

Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which complicate quantization to low bit-widths. We introduce BitNet v2, a novel framework enabling native 4-bit activation quantization for 1-bit LLMs. To tackle outliers in attention and feed-forward network activations, we propose H-BitLinear, a module applying an online Hadamard transformation prior to activation quantization. This transformation smooths sharp activation distributions into more Gaussian-like forms, suitable for low-bit representation. Experiments show BitNet v2 trained from scratch with 8-bit activations matches BitNet b1.58 performance. Crucially, BitNet v2 achieves minimal performance degradation when trained with native 4-bit activations, significantly reducing memory footprint and computational cost for batched inference.