Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Not on the map. Why?
Abstract (arXiv)
Sink-aware deployment may identify important first-token attention heads before a model is quantized, then reuse that map at the edge. We test when this shortcut is safe for 4-bit NF4 weight-only post-training quantization (PTQ). Our Sink Topology Consistency (STC) metrics separate global rank preservation, top-k set overlap, and layerwise sink-mass shift, and distinguish per-input sensitivity from calibration-map transfer. Across Qwen2.5-0.5B, Qwen2.5-1.5B, and Llama-3.2-1B, global bf16-to-4-bit ranks remain high at 4,096 tokens (ρs ≥ 0.980), yet top-k Jaccard overlap is only 0.619-0.793, corresponding to 76.5-88.5% membership retention. The global statistic also masks local failures: terminal Qwen layers shift by 6.2-7.9x their model means, whereas Llama-3.2-1B shows low, nearly uniform drift. Under a C4-to-LongBench shift, cross-domain overlap degrades more than the within-domain precision comparison for both Qwen models, but not for Llama-3.2-1B. Matched-domain 4-bit recalibration reaches 90% of a split-half stability plateau at the smallest tested n=8 for both Qwen models and n=32 for Llama-3.2-1B, though not as a sharp threshold; for the two Qwen models, updating only selected layers does not reach the full-map stability criterion. On Jetson Orin NX, the 16-sample workload takes seconds for the two models with valid on-device sink measurements. The practical message is precise: global rankings often transfer, but discrete head sets, layer-local policies, and cross-domain calibration should be revalidated after quantization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 13 references that are in the map. First posted on arXiv 20 Sep 2026.
Builds on
13 of 13 references.
References · 13
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2024Massive Activations in Large Language Modelscited by 294
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
- 2025Why do LLMs attend to the first token?cited by 127
- 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
- 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
- 2025What are you sinking? A geometric approach on attention sinkcited by 28
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence
- 2026Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
- 2026Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
- 2026From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
- 2026Normalized Architectures are Natively 4-Bit
- 2026STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
- 2026RAMP: Robust Adaptive Mixed-Precision Quantization for Edge CPU Vision Models
- 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedom