mlsanityData to 7 Oct 2026

The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

arXiv 2605.20749 ↗First posted 20 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution. Building on this finding, we further analyze the resulting training dynamics and show how the reshaped spectrum leads to faster convergence of GLU models, including a characteristic loss-crossing phenomenon observed between GLU and non-GLU models. Finally, we empirically observe that GLU has limited impact in reducing the generalization gap on various models, including ViT and GPT-2, suggesting that its primary benefit lies in accelerating optimization rather than reducing the generalization gap. The code is available at: https://github.com/Zemdalk/GLU-NTK.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

28 of 51 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2016Language Modeling with Gated Convolutional Networkscited by 688
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2025Kimi K2: Open Agentic Intelligencecited by 523
16 more
  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  2. 2015Highway Networkscited by 257
  3. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  4. 2016Exploring the Limits of Language Modelingcited by 172
  5. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  6. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  7. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  8. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  9. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  10. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  11. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  12. 2024Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regimecited by 11
  13. 2023Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networkscited by 6
  14. 2024SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAMcited by 3
  15. 2025GLU Attention Improve Transformercited by 3
  16. 2025MLPs at the EOC: Spectrum of the NTKcited by 1

Built on by

Not cited yet.