# One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

arXiv: https://arxiv.org/abs/2608.02091 · HTML: https://mlsanity.com/paper/2608.02091/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Online normalizer calculation for softmax](https://mlsanity.com/paper/1805.02867/) (2018)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [Oscillation-free Quantization for Low-bit Vision Transformers](https://mlsanity.com/paper/2302.02210/) (2023)
- [A Theory on Adam Instability in Large-Scale Machine Learning](https://mlsanity.com/paper/2304.09871/) (2023)
- [SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training](https://mlsanity.com/paper/2501.06842/) (2025)
- [Methods of improving LLM training stability](https://mlsanity.com/paper/2410.16682/) (2024)
- [Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention](https://mlsanity.com/paper/2510.04212/) (2025)
- [Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks](https://mlsanity.com/paper/1901.06588/) (2019)
- [Taming Transformer Without Using Learning Rate Warmup](https://mlsanity.com/paper/2505.21910/) (2025)
- [AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping](https://mlsanity.com/paper/2502.11034/) (2025)
- [Pretraining large language models with MXFP4 on Native FP4 Hardware](https://mlsanity.com/paper/2605.09825/) (2026)
- [Characterization and Mitigation of Training Instabilities in Microscaling Formats](https://mlsanity.com/paper/2506.20752/) (2025)
- [Controlling changes to attention logits](https://mlsanity.com/paper/2511.21377/) (2025)
- [Spectral Alignment as Predictor of Loss Explosion in Neural Network Training](https://mlsanity.com/paper/2510.04202/) (2025)
- [Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training](https://mlsanity.com/paper/2602.18851/) (2026)
- [Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability](https://mlsanity.com/paper/2606.28116/) (2026)

## Abstract (arXiv)

A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source.