# A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attention

arXiv: https://arxiv.org/abs/2507.09394 · HTML: https://mlsanity.com/paper/2507.09394/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 12 Jul 2025.

## Builds on (most cited first, 15 of 15)

- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize](https://mlsanity.com/paper/2203.06176/) (2022)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN](https://mlsanity.com/paper/2412.13795/) (2024)
- [TransMLA: Multi-Head Latent Attention Is All You Need](https://mlsanity.com/paper/2502.07864/) (2025)
- [A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning](https://mlsanity.com/paper/1912.00827/) (2019)
- [Self-attention Networks Localize When QK-eigenspectrum Concentrates](https://mlsanity.com/paper/2402.02098/) (2024)
- [A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities](https://mlsanity.com/paper/2410.18938/) (2024)
- [The Dynamics of Learning: A Random Matrix Approach](https://mlsanity.com/paper/1805.11917/) (2018)
- [The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets](https://mlsanity.com/paper/2306.14975/) (2023)
- [Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting](https://mlsanity.com/paper/2406.10327/) (2024)
- [Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory](https://mlsanity.com/paper/2410.08942/) (2024)
- [Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption](https://mlsanity.com/paper/2310.13434/) (2023)
- [Random Matrix Improved Covariance Estimation for a Large Class of Metrics](https://mlsanity.com/paper/1902.02554/) (2019)

## Built on by (most cited first, 1 of 1)

- [How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models](https://mlsanity.com/paper/2605.06826/) (2026)

## Abstract (arXiv)

In this work, we study how multi-head latent attention (MLA), a popular strategy for compressing key/value memory, affects a transformer's internal capacity during pretraining. Using a lightweight suite of Marchenko-Pastur (MP) diagnostics, we analyze the spectrum of the WQWK^top gram matrix throughout training, comparing three variants: the standard multi-head attention (MHA) baseline, MLA-PreRoPE with rotary applied before compression, and MLA-Decoupled, which shares a single rotary sub-vector across all heads. Our random matrix analysis reveals three key findings:  i) capacity bottlenecks emerge locally: both MHA and MLA-PreRoPE exhibit sharp, early spikes in specific layers that persist and propagate, disrupting the balance between bulk and outlier directions;  ii) these spikes coincide with rank collapse, concentrating the model's expressivity into narrow subspaces;  iii) only the decoupled variant prevents this cascade, maintaining broad spectral support and suppressing outlier formation across layers. These results underscore that how rotary embeddings are applied is just as critical as where compression occurs. Sharing rotary components across heads mitigates spectral fragmentation and preserves representational capacity.