mlsanityData to 7 Oct 2026

A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio

arXiv 2605.28975 ↗First posted 27 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the log-alignment ratio (LAR), a measure of parameter-activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectrum p of the normalized squared singular values of a matrix and an activation spectrum q of the normalized squared projections of inputs onto its singular directions. We show that unembedding LAR tracks the transition between memorization and generalization in two different settings by capturing the spread of p and q during training. In grokking, LAR predicts the effective dimension of the learned function: k ≈ n2(1-LAR), where n is the input dimension of the matrix. In 3B-parameter language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap, and its rate of decline increases as overfitting approaches. LAR is computable from quantities available during the forward pass with negligible computational overhead, and requires no held-out validation data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

25 of 31 references.

References · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  10. 2025Muon is Scalable for LLM Trainingcited by 381
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  12. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
13 more
  1. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  2. 2016Using the Output Embedding to Improve Language Modelscited by 266
  3. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  4. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  5. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  6. 2023Explaining grokking through circuit efficiencycited by 105
  7. 2023A Spectral Condition for Feature Learningcited by 101
  8. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 71
  9. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  10. 2023Unifying Grokking and Double Descentcited by 49
  11. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  12. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  13. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
  2. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?
  3. 2026Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models
  4. 2025Understanding Post-Training Structural Changes in Large Language Models
  5. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test
  6. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
  7. 2025Characterizing Pattern Matching and Its Limits on Compositional Task Structures
  8. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling