Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training

Attention scores in transformers are bilinear forms Sij = xi^top M xj / √dh whose maximum magnitude governs overflow risk in low-precision training. We derive a rank-aware concentration inequality: when the interaction matrix M = WQ WKtop has rank r ll d, tail probabilities for maxi,j|Sij| decay as exp(-d2α2/(γ r)) rather than exp(-dα2), where γ > 1 is a typicality parameter. For transformer attention where r = dh, this yields 8--28× tighter concentration than rank-agnostic bounds in modern architectures. We apply this result to FP8 training, deriving geometry-aware scale factors that provide principled overflow guarantees without observing activations. The method computes per-layer scales from the spectral norm |WQ WKtop|2 via implicit power iteration, includes a grouped query attention formulation that avoids key expansion, and remains compatible with fused attention kernels. Across GPT-2 XL to Llama-2-70B, geometry-aware scaling eliminates overflows in transient scenarios where delayed scaling fails, while achieving comparable downstream MMLU accuracy.