# On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks

arXiv: https://arxiv.org/abs/2410.08041 · HTML: https://mlsanity.com/paper/2410.08041/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 65 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation](https://mlsanity.com/paper/2406.02918/) (2024)
- [Wav-KAN: Wavelet Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.12832/) (2024)
- [Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation](https://mlsanity.com/paper/2405.07200/) (2024)
- [KAN 2.0: Kolmogorov-Arnold Networks Meet Science](https://mlsanity.com/paper/2408.10205/) (2024)
- [DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems](https://mlsanity.com/paper/2405.19143/) (2024)
- [Convolutional Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2406.13155/) (2024)
- [KAN or MLP: A Fairer Comparison](https://mlsanity.com/paper/2407.16674/) (2024)
- [TKAN: Temporal Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.07344/) (2024)
- [Kolmogorov-Arnold Networks for Time Series: Bridging Predictive Power and Interpretability](https://mlsanity.com/paper/2406.02496/) (2024)
- [Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks](https://mlsanity.com/paper/2406.11045/) (2024)
- [fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions](https://mlsanity.com/paper/2406.07456/) (2024)
- [A Temporal Kolmogorov-Arnold Transformer for Time Series Forecasting](https://mlsanity.com/paper/2406.02486/) (2024)
- [Demonstrating the Efficacy of Kolmogorov-Arnold Networks in Vision Tasks](https://mlsanity.com/paper/2406.14916/) (2024)
- [Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.17611/) (2024)
- [Exploring the Potential of Polynomial Basis Functions in Kolmogorov-Arnold Networks: A Comparative Study of Different Groups of Polynomials](https://mlsanity.com/paper/2406.02583/) (2024)
- [rKAN: Rational Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2406.14495/) (2024)
- [Finite basis Kolmogorov-Arnold networks: domain decomposition for data-driven and physics-informed problems](https://mlsanity.com/paper/2406.19662/) (2024)
- [ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition, Dot Multiplication, and ReLU](https://mlsanity.com/paper/2406.02075/) (2024)
- [GraphKAN: Enhancing Feature Extraction with Graph Kolmogorov Arnold Networks](https://mlsanity.com/paper/2406.13597/) (2024)
- [No bad local minima: Data independent training error guarantees for multilayer neural networks](https://mlsanity.com/paper/1605.08361/) (2016)
- [Kolmogorov-Arnold Convolutions: Design Principles and Empirical Studies](https://mlsanity.com/paper/2407.01092/) (2024)
- [Kolmogorov-Arnold Networks: A Critical Assessment of Claims, Performance, and Practical Viability](https://mlsanity.com/paper/2407.11075/) (2024)
- [CoxKAN: Kolmogorov-Arnold Networks for Interpretable, High-Performance Survival Analysis](https://mlsanity.com/paper/2409.04290/) (2024)
- [The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.03864/) (2024)
- [FC-KAN: Function Combinations in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2409.01763/) (2024)
- [Inferring turbulent velocity and temperature fields and their statistics from Lagrangian velocity measurements using physics-informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.15727/) (2024)
- [Kolmogorov-Arnold Network for Online Reinforcement Learning](https://mlsanity.com/paper/2408.04841/) (2024)
- [Gaussian Process Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.18397/) (2024)

## Abstract (arXiv)

Kolmogorov--Arnold Networks (KANs), a recently proposed neural network architecture, have gained significant attention in the deep learning community, due to their potential as a viable alternative to multi-layer perceptrons (MLPs) and their broad applicability to various scientific tasks. Empirical investigations demonstrate that KANs optimized via stochastic gradient descent (SGD) are capable of achieving near-zero training loss in various machine learning (e.g., regression, classification, and time series forecasting, etc.) and scientific tasks (e.g., solving partial differential equations). In this paper, we provide a theoretical explanation for the empirical success by conducting a rigorous convergence analysis of gradient descent (GD) and SGD for two-layer KANs in solving both regression and physics-informed tasks. For regression problems, we establish using the neural tangent kernel perspective that GD achieves global linear convergence of the objective function when the hidden dimension of KANs is sufficiently large. We further extend these results to SGD, demonstrating a similar global convergence in expectation. Additionally, we analyze the global convergence of GD and SGD for physics-informed KANs, which unveils additional challenges due to the more complex loss structure. This is the first work establishing the global convergence guarantees for GD and SGD applied to optimize KANs and physics-informed KANs.