# Bitnet.cpp: Efficient Edge Inference for Ternary LLMs

arXiv: https://arxiv.org/abs/2502.11880 · HTML: https://mlsanity.com/paper/2502.11880/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models](https://mlsanity.com/paper/2206.09557/) (2022)
- [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://mlsanity.com/paper/2406.06282/) (2024)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models](https://mlsanity.com/paper/2310.16795/) (2023)
- [Multiplying Matrices Without Multiplying](https://mlsanity.com/paper/2106.10860/) (2021)
- [LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup](https://mlsanity.com/paper/2302.03213/) (2023)
- [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://mlsanity.com/paper/2411.04965/) (2024)
- [1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://mlsanity.com/paper/2410.16144/) (2024)
- [DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables](https://mlsanity.com/paper/2304.09049/) (2023)

## Built on by (most cited first, 22 of 22)

- [BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation](https://mlsanity.com/paper/2506.07530/) (2025)
- [Tequila: Trapping-free Ternary Quantization for Large Language Models](https://mlsanity.com/paper/2509.23809/) (2025)
- [TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge](https://mlsanity.com/paper/2509.13765/) (2025)
- [Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification](https://mlsanity.com/paper/2601.07892/) (2026)
- [Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash](https://mlsanity.com/paper/2504.08378/) (2025)
- [PT2-LLM: Post-Training Ternarization for Large Language Models](https://mlsanity.com/paper/2510.03267/) (2025)
- [DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones](https://mlsanity.com/paper/2511.07427/) (2025)
- [T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization](https://mlsanity.com/paper/2511.13676/) (2025)
- [vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models](https://mlsanity.com/paper/2606.08094/) (2026)
- [Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplication](https://mlsanity.com/paper/2511.21910/) (2025)
- [Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± i](https://mlsanity.com/paper/2512.02901/) (2025)
- [Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices](https://mlsanity.com/paper/2512.06443/) (2025)
- [BitHEP -- The Limits of Low-Precision ML in HEP](https://mlsanity.com/paper/2504.03387/) (2025)
- [Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models](https://mlsanity.com/paper/2508.06753/) (2025)
- [Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs](https://mlsanity.com/paper/2606.03026/) (2026)
- [Breaking the 1.58-bit Barrier for Ternary LLMs](https://mlsanity.com/paper/2609.16338/) (2026)
- [MAGNET: Autonomous Expert Model Generation via Decentralized Autoresearch and BitNet Training](https://mlsanity.com/paper/2603.25813/) (2026)
- [PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference](https://mlsanity.com/paper/2607.14618/) (2026)
- [Ankhdjet: An Open-Source Compiler for Mask-Programmed Ternary Compute-in-ROM on an Open PDK](https://mlsanity.com/paper/2608.26206/) (2026)
- [QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization](https://mlsanity.com/paper/2609.00224/) (2026)
- [Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions](https://mlsanity.com/paper/2609.27453/) (2026)
- [SSR: Sparse Segment Reduction for Ternary GEMM Acceleration](https://mlsanity.com/paper/2610.08403/) (2026)

## Abstract (arXiv)

The advent of 1-bit large language models (LLMs), led by BitNet b1.58, has spurred interest in ternary LLMs. Despite this, research and practical applications focusing on efficient edge inference for ternary LLMs remain scarce. To bridge this gap, we introduce Bitnet.cpp, an inference system optimized for BitNet b1.58 and ternary LLMs. Given that mixed-precision matrix multiplication (mpGEMM) constitutes the bulk of inference time in ternary LLMs, Bitnet.cpp incorporates a novel mpGEMM library to facilitate sub-2-bits-per-weight, efficient and lossless inference. The library features two core solutions: Ternary Lookup Table (TL), which addresses spatial inefficiencies of previous bit-wise methods, and Int2 with a Scale (I2_S), which ensures lossless edge inference, both enabling high-speed inference. Our experiments show that Bitnet.cpp achieves up to a 6.25x increase in speed over full-precision baselines and up to 2.32x over low-bit baselines, setting new benchmarks in the field. Additionally, we expand TL to element-wise lookup table (ELUT) for low-bit LLMs in the appendix, presenting both theoretical and empirical evidence of its considerable potential. Bitnet.cpp is publicly available at https://github.com/microsoft/BitNet/tree/paper , offering a sophisticated solution for the efficient and practical deployment of edge LLMs.