# Breaking the 1.58-bit Barrier for Ternary LLMs

arXiv: https://arxiv.org/abs/2609.16338 · HTML: https://mlsanity.com/paper/2609.16338/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [Multiplying Matrices Without Multiplying](https://mlsanity.com/paper/2106.10860/) (2021)
- [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://mlsanity.com/paper/2502.11880/) (2025)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)
- [LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup](https://mlsanity.com/paper/2302.03213/) (2023)
- [Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization](https://mlsanity.com/paper/2311.16442/) (2023)
- [Tequila: Trapping-free Ternary Quantization for Large Language Models](https://mlsanity.com/paper/2509.23809/) (2025)
- [DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables](https://mlsanity.com/paper/2304.09049/) (2023)
- [TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization](https://mlsanity.com/paper/2606.13054/) (2026)
- [Binary and Ternary Natural Language Generation](https://mlsanity.com/paper/2306.01841/) (2023)
- [CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs](https://mlsanity.com/paper/2606.26650/) (2026)
- [Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification](https://mlsanity.com/paper/2601.07892/) (2026)
- [Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity](https://mlsanity.com/paper/2603.05168/) (2026)
- [Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Models](https://mlsanity.com/paper/2508.06753/) (2025)

## Built on by (most cited first, 1 of 1)

- [Behavioral Capacity Certificates for Quantized Language Models](https://mlsanity.com/paper/2609.37887/) (2026)

## Abstract (arXiv)

Ternary Large Language Models (LLM) store every weight as one of three symbols -1,0,+1, so the cost of a ternary model is conventionally referenced to the information-theoretic log2 3 ≈ 1.585 bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to 1.625 bits per weight. This effective storage bit-width treats the three symbols -1,0,+1 as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to 51.5% of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs 2 - z bits per weight element given a zero density z in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches 1.485 bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to 1.28×. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to 1.18× on CPUs and 1.27× on GPUs.