mlsanityData to 14 Oct 2026

Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair

arXiv 2608.23144 ↗First posted 24 Aug 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Low-bit weight quantization saves storage but leaves errors that degrade LLM quality. We introduce activation-weighted seeded residual coding (AWSRC), a compact repair codec for an existing quantization backbone. Given a reconstructed weight W0, AWSRC encodes the residual W-W0 using deterministic seed-generated bases. The sidecar stores seed selectors, low-bit coefficients, and scales rather than an explicit codebook. Two variants combine activation weighting with per-module byte quotas (mathrmAWSRC-U), or blended activation/Fisher weighting with globally ranked progressive prefixes (mathrmAWSRC-PF) that support multiple byte budgets without refitting. On Qwen2.5-3B-Instruct, adding 0.162 scope-bits/weight to an RTN-INT4 baseline closes 88.2%, 78.9%, and 71.3% of the PPL, KL, and 11-task mean-accuracy gaps to BF16, respectively. AWSRC achieves the highest mean downstream accuracy in byte-matched residual-codec ablations and improves all metrics across model families with up to 32B parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

26 of 30 references.

References · 26

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023Mistral 7Bcited by 3,164
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
14 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  7. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  8. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  9. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  10. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  11. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  12. 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environmentscited by 9
  13. 2024SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generatorscited by 4
  14. 2026QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scalingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
  2. 2026Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
  3. 2025CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
  4. 2026Technical Report: Activation Residual Hessian Quantization (ARHQ) for Low-Bit LLM Quantization
  5. 2026Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals
  6. 2026Transforms for LLM Quantization: The Great Inversion and Format Co-Design
  7. 2026BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization
  8. 2026CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights