mlsanityData to 14 Oct 2026

QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization

arXiv 2609.00224 ↗First posted 31 Aug 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy loss below 2 bits. Many leverage unstructured sparsity to mitigate this loss, but at the cost of regularity and GPU-friendly execution. We present QTEA, a sub-2-bit PTQ framework that quantizes weights into ternary values and uses salient weights as residual error compensators. To maintain hardware efficiency, residuals are assigned to selected columns with semi-structured 1:4 sparsity within the salient columns. We further add column-wise rescale refinement to GPTQ-style column-by-column quantization, alternately updating per-column scales and ternary assignments to reduce reconstruction error. We also identify order-dependent error propagation in GPTQ and introduce error decay to attenuate late-stage error accumulation. On Qwen3-14B, QTEA compresses all weights to an effective 1.7 bits per weight while improving average accuracy over the strongest ternary PTQ baseline by 16.7%. It also achieves 1.40× and 2.61× lower perplexity on WikiText and C4 respectively. This trend holds on Llama3-8B, where QTEA obtains a 6.6% accuracy gain and 1.34× / 1.95× lower perplexity on the same datasets. Finally, we develop a lookup-table based kernel that achieves 7.2× faster per-token generation over an FP16 baseline. Code is available at https://github.com/Intelligent-Microsystems-Lab/QTEA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

21 of 31 references.

References · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
9 more
  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  2. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  3. 2020Sparse GPU Kernels for Deep Learningcited by 141
  4. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  5. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  6. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 22
  7. 2023ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformercited by 15
  8. 2024GPTQT: Quantize Large Language Models Twice to Push the Efficiencycited by 4
  9. 2024ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
  2. 2024CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
  3. 2026TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
  4. 2025PT2-LLM: Post-Training Ternarization for Large Language Models
  5. 2026G2PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation
  6. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
  7. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Models
  8. 2025SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights