mlsanityData to 7 Oct 2026

ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

arXiv 2608.02703 ↗First posted 3 Aug 2026cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

14 of 23 references.

References · 14

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
2 more
  1. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  2. 2022Language model compression with weighted low-rank factorizationcited by 192

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Softmax Reparameterization for Output-Head Quantizationcited by 0