# Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

arXiv: https://arxiv.org/abs/2505.15548 · HTML: https://mlsanity.com/paper/2505.15548/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Low-Rank Bottleneck in Multi-head Attention Models](https://mlsanity.com/paper/2002.07028/) (2020)
- [Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models](https://mlsanity.com/paper/2403.09635/) (2024)
- [Zero-Shot Styled Text Image Generation, but Make It Autoregressive](https://mlsanity.com/paper/2503.17074/) (2025)

## Built on by (most cited first, 2 of 2)

- [SymLoc: Symbolic Localization of Hallucination across HaluEval and TruthfulQA](https://mlsanity.com/paper/2511.14172/) (2025)
- [QUEST: A robust attention formulation using query-modulated spherical attention](https://mlsanity.com/paper/2604.00199/) (2026)

## Abstract (arXiv)

Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit explosion, its underlying cause remains poorly understood. In this work, we present analytical insights and empirical evidence that dense local dependencies are a major contributor to attention-logit explosion. We demonstrate that dense local dependency patterns yield an effectively high-rank attention structure, which the low-rank parameterization of self-attention can only approximate with increasingly large logits as the sequence length grows. This logit inflation ultimately leads to training instability under low-precision arithmetic. We support this explanation through extensive experiments on synthetic and language modeling tasks. Our results consistently show that attention-logit growth increases with sequence length, is mitigated by increasing the attention dimension, and is substantially reduced by explicitly modeling dense local dependencies. Furthermore, we show that this growth is driven by the density of local dependencies rather than by locality alone. More broadly, our findings suggest that explicitly modeling dense local dependencies constitutes an important design principle for developing stable, efficient, and scalable long-context transformer architectures for autoregressive language modeling.