mlsanityData to 14 Oct 2026

Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition

arXiv 2604.18128 ↗First posted 20 Apr 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 to 1727. A simple residual-axis training-time intervention -- Depth Registers with a register-magnitude hinge loss (DR+sink) -- reduces this to 119 (about 14x) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 PPL. The residual ~2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers (qkv, w1, w3) and block-internal generators (o_proj, w2). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2's bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ~0.28 nats across three matched checkpoints with Delta-remove(w2) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot -- adding online per-head value Hadamard plus online w2-input rotation -- does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

18 of 20 references.

References · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Mistral 7Bcited by 3,164
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2023Vision Transformers Need Registerscited by 871
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
6 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  3. 2024Massive Activations in Large Language Modelscited by 294
  4. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  5. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  6. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026W4A4 Quantization for Inference on Wan2.2-I2V-A14B
  2. 2026Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4
  3. 2026ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
  4. 2026QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization
  5. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases
  6. 2026Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals
  7. 2026When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence
  8. 2026From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization