mlsanityData to 14 Oct 2026

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

arXiv 2609.35465 ↗First posted 28 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above llama.cpp's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 25 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

17 of 25 references.

References · 17

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  11. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
5 more
  1. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  2. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  3. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  4. 2026Leech Lattice Vector Quantization for Efficient LLM Compressioncited by 7
  5. 2026Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weightscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026BCJR-QAT: A Differentiable Relaxation of Trellis-Coded Weight Quantization
  2. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
  3. 2025Spherical Leech Quantization for Visual Tokenization and Generation
  4. 2024QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering
  5. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
  6. 2025AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
  7. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Design
  8. 2022FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs