# Spectral Alignment as Predictor of Loss Explosion in Neural Network Training

arXiv: https://arxiv.org/abs/2510.04202 · HTML: https://mlsanity.com/paper/2510.04202/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [DeepNet: Scaling Transformers to 1,000 Layers](https://mlsanity.com/paper/2203.00555/) (2022)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [A Spectral Condition for Feature Learning](https://mlsanity.com/paper/2310.17813/) (2023)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)
- [Spike No More: Stabilizing the Pre-training of Large Language Models](https://mlsanity.com/paper/2312.16903/) (2023)
- [Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks](https://mlsanity.com/paper/1806.05394/) (2018)
- [Stable Rank Normalization for Improved Generalization in Neural Networks and GANs](https://mlsanity.com/paper/1906.04659/) (2019)
- [projUNN: efficient method for training deep networks with unitary matrices](https://mlsanity.com/paper/2203.05483/) (2022)
- [Sharpness-Aware Minimization and the Edge of Stability](https://mlsanity.com/paper/2309.12488/) (2023)
- [Methods of improving LLM training stability](https://mlsanity.com/paper/2410.16682/) (2024)
- [Representation Alignment in Neural Networks](https://mlsanity.com/paper/2112.07806/) (2021)
- [BranchNorm: Robustly Scaling Extremely Deep Transformers](https://mlsanity.com/paper/2305.02790/) (2023)
- [Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes](https://mlsanity.com/paper/2410.05052/) (2024)

## Built on by (most cited first, 2 of 2)

- [One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse](https://mlsanity.com/paper/2608.02091/) (2026)
- [Backward-State Policy Is Part of the Learning Algorithm](https://mlsanity.com/paper/2609.39813/) (2026)

## Abstract (arXiv)

Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging and ambiguous predictors, as their values vary dramatically across different models and even between layers of the same model, making it difficult to establish a unified standard for detecting impending failure. We introduce Spectral Alignment (SA), a novel, theoretically-grounded metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices. We show that a collapse in the sign diversity of this alignment is a powerful early predictor of representational collapse and training divergence. Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics. SA's low computational overhead makes it a practical tool for safeguarding model training.