# ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition, Dot Multiplication, and ReLU

arXiv: https://arxiv.org/abs/2406.02075 · HTML: https://mlsanity.com/paper/2406.02075/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 4 of 4)

- [Wav-KAN: Wavelet Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.12832/) (2024)
- [Kolmogorov-Arnold Networks (KANs) for Time Series Analysis](https://mlsanity.com/paper/2405.08790/) (2024)
- [DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems](https://mlsanity.com/paper/2405.19143/) (2024)
- [TKAN: Temporal Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.07344/) (2024)

## Built on by (most cited first, 30 of 33)

- [Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.17611/) (2024)
- [Finite basis Kolmogorov-Arnold networks: domain decomposition for data-driven and physics-informed problems](https://mlsanity.com/paper/2406.19662/) (2024)
- [FC-KAN: Function Combinations in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2409.01763/) (2024)
- [Reduced Effectiveness of Kolmogorov-Arnold Networks on Functions with Noise](https://mlsanity.com/paper/2407.14882/) (2024)
- [Higher-order-ReLU-KANs (HRKANs) for solving physics-informed neural networks (PINNs) more accurately, robustly and faster](https://mlsanity.com/paper/2409.14248/) (2024)
- [Activation Space Selectable Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2408.08338/) (2024)
- [Multifidelity Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2410.14764/) (2024)
- [KAN versus MLP on Irregular or Noisy Functions](https://mlsanity.com/paper/2408.07906/) (2024)
- [FEKAN: Feature-Enriched Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.16530/) (2026)
- [Introducing the Short-Time Fourier Kolmogorov Arnold Network: A Dynamic Graph CNN Approach for Tree Species Classification in 3D Point Clouds](https://mlsanity.com/paper/2503.23647/) (2025)
- [ChemKANs for Combustion Chemistry Modeling and Acceleration](https://mlsanity.com/paper/2504.12580/) (2025)
- [Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement](https://mlsanity.com/paper/2505.18131/) (2025)
- [KAN v.s. MLP for Offline Reinforcement Learning](https://mlsanity.com/paper/2409.09653/) (2024)
- [P1-KAN: an effective Kolmogorov-Arnold network with application to hydraulic valley optimization](https://mlsanity.com/paper/2410.03801/) (2024)
- [Evaluating Federated Kolmogorov-Arnold Networks on Non-IID Data](https://mlsanity.com/paper/2410.08961/) (2024)
- [AF-KAN: Activation Function-Based Kolmogorov-Arnold Networks for Efficient Representation Learning](https://mlsanity.com/paper/2503.06112/) (2025)
- [Automatic Grid Updates for Kolmogorov-Arnold Networks using Layer Histograms](https://mlsanity.com/paper/2511.08570/) (2025)
- [Making Gaussian Kolmogorov-Arnold Networks Reliable and Accurate](https://mlsanity.com/paper/2604.21174/) (2026)
- [Kolmogorov Arnold Networks in Fraud Detection: Bridging the Gap Between Theory and Practice](https://mlsanity.com/paper/2408.10263/) (2024)
- [SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions](https://mlsanity.com/paper/2607.18290/) (2026)
- [Uncertainty Quantification with Bayesian Higher Order ReLU KANs](https://mlsanity.com/paper/2410.01687/) (2024)
- [Scaled Supervision is an Implicit Lipschitz Regularizer](https://mlsanity.com/paper/2503.14813/) (2025)
- [Sinusoidal Approximation Theorem for Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2508.00247/) (2025)
- [Multilevel Training for Kolmogorov Arnold Networks](https://mlsanity.com/paper/2603.04827/) (2026)
- [Partition-of-Unity Gaussian Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2604.23599/) (2026)
- [RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis](https://mlsanity.com/paper/2609.01729/) (2026)
- [On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks](https://mlsanity.com/paper/2410.08041/) (2024)
- [HKAN: Hierarchical Kolmogorov-Arnold Network without Backpropagation](https://mlsanity.com/paper/2501.18199/) (2025)
- [Combinations of Fast Activation and Trigonometric Functions in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2508.11876/) (2025)
- [PolyKAN: A Polyhedral Analysis Framework for Provable and Approximately Optimal KAN Compression](https://mlsanity.com/paper/2510.04205/) (2025)

## Abstract (arXiv)

Limited by the complexity of basis function (B-spline) calculations, Kolmogorov-Arnold Networks (KAN) suffer from restricted parallel computing capability on GPUs. This paper proposes a novel ReLU-KAN implementation that inherits the core idea of KAN. By adopting ReLU (Rectified Linear Unit) and point-wise multiplication, we simplify the design of KAN's basis function and optimize the computation process for efficient CUDA computing. The proposed ReLU-KAN architecture can be readily implemented on existing deep learning frameworks (e.g., PyTorch) for both inference and training. Experimental results demonstrate that ReLU-KAN achieves a 20x speedup compared to traditional KAN with 4-layer networks. Furthermore, ReLU-KAN exhibits a more stable training process with superior fitting ability while preserving the "catastrophic forgetting avoidance" property of KAN. You can get the code in https://github.com/quiqi/relu_kan