# BitNet b1.58 2B4T Technical Report

arXiv: https://arxiv.org/abs/2504.12285 · HTML: https://mlsanity.com/paper/2504.12285/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [MathScale: Scaling Instruction Tuning for Mathematical Reasoning](https://mlsanity.com/paper/2403.02884/) (2024)

## Built on by (most cited first, 30 of 38)

- [BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation](https://mlsanity.com/paper/2506.07530/) (2025)
- [Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees](https://mlsanity.com/paper/2505.18659/) (2025)
- [PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models](https://mlsanity.com/paper/2509.16989/) (2025)
- [Tequila: Trapping-free Ternary Quantization for Large Language Models](https://mlsanity.com/paper/2509.23809/) (2025)
- [Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs](https://mlsanity.com/paper/2509.11177/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression](https://mlsanity.com/paper/2602.21233/) (2026)
- [iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± i](https://mlsanity.com/paper/2508.05571/) (2025)
- [Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification](https://mlsanity.com/paper/2601.07892/) (2026)
- [Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization](https://mlsanity.com/paper/2608.01078/) (2026)
- [DP-Adam-AC: Privacy-preserving Fine-Tuning of Localizable Language Models Using Adam Optimization with Adaptive Clipping](https://mlsanity.com/paper/2510.05288/) (2025)
- [Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs](https://mlsanity.com/paper/2604.07888/) (2026)
- [LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization](https://mlsanity.com/paper/2606.10531/) (2026)
- [BitNet Distillation](https://mlsanity.com/paper/2510.13998/) (2025)
- [Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± i](https://mlsanity.com/paper/2512.02901/) (2025)
- [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices](https://mlsanity.com/paper/2512.06443/) (2025)
- [COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection](https://mlsanity.com/paper/2505.17701/) (2025)
- [Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models](https://mlsanity.com/paper/2508.06974/) (2025)
- [BitSkip: An Empirical Analysis of Quantization and Early Exit Composition in Transformers](https://mlsanity.com/paper/2510.23766/) (2025)
- [TOM: A Ternary Read-only Memory Accelerator for LLM-powered Edge Intelligence](https://mlsanity.com/paper/2602.20662/) (2026)
- [VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devices](https://mlsanity.com/paper/2605.00320/) (2026)
- [Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs](https://mlsanity.com/paper/2606.03026/) (2026)
- [When Can Depth Replace Precision? A Resource Theory of Quantized Neural Computation](https://mlsanity.com/paper/2607.23390/) (2026)
- [Breaking the 1.58-bit Barrier for Ternary LLMs](https://mlsanity.com/paper/2609.16338/) (2026)
- [TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edge](https://mlsanity.com/paper/2609.33548/) (2026)
- [Transforming Decoder-Only Transformers for Accurate WiFi-Telemetry Based Indoor Localization](https://mlsanity.com/paper/2505.15835/) (2025)
- [Ultra-Quantisation: Efficient Embedding Search via 1.58-bit Encodings](https://mlsanity.com/paper/2506.00528/) (2025)
- [Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions](https://mlsanity.com/paper/2510.22909/) (2025)
- [pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training](https://mlsanity.com/paper/2602.22592/) (2026)
- [On the Expressive Power of Weight Quantization in Large Language Models](https://mlsanity.com/paper/2606.22249/) (2026)

## Abstract (arXiv)

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.