mlsanityData to 14 Oct 2026

The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression

arXiv 2604.07380 ↗First posted 8 Apr 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We decompose the spectral edge -- the dominant direction of the Gram matrix of parameter updates -- into its gradient and weight-decay components during grokking in two sequence tasks (Dyck-1 and SCAN). We find a sharp two-phase lifecycle: before grokking the edge is gradient-driven and functionally active; at grokking, gradient and weight decay align, and the edge becomes a compression axis that is perturbation-flat yet ablation-critical (>4000x more impactful than random directions). Three universality classes emerge (functional, mixed, compression), predicted by the gap flow equation. Nonlinear probes show information is re-encoded, not lost (MLP R2=0.99 where linear R2=0.86), and removing weight decay post-grok reverses compression while preserving the algorithm.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

14 of 24 references.

References · 14

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  5. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  6. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  7. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  8. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 80
  9. 2026Low-Dimensional and Transversely Curved Optimization Dynamics in Grokkingcited by 11
  10. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 9
  11. 2026Low-Dimensional Execution Manifolds in Transformer Learning Dynamics: Evidence from Modular Arithmetic Taskscited by 7
  12. 2026Early-Warning Signals of Grokking via Loss-Landscape Geometrycited by 7
2 more
  1. 2026Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Trainingcited by 4
  2. 2026Spectral Edge Dynamics Reveal Functional Modes of Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-training
  2. 2026Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
  3. 2025Constrained Edge AI Deployment: Fine-Tuning vs Distillation for LLM Compression
  4. 2025Data Curation Through the Lens of Spectral Dynamics: Static Limits, Dynamic Acceleration, and Practical Oracles
  5. 2026Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
  6. 2026Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
  7. 2026What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle
  8. 2026FuncCode: Compressing Kolmogorov--Arnold Networks in Function Space with Hardware-Aware Quantization