# Kolmogorov-Arnold Transformer

arXiv: https://arxiv.org/abs/2409.10594 · HTML: https://mlsanity.com/paper/2409.10594/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Kolmogorov-Arnold Networks are Radial Basis Function Networks](https://mlsanity.com/paper/2405.06721/) (2024)
- [KAN 2.0: Kolmogorov-Arnold Networks Meet Science](https://mlsanity.com/paper/2408.10205/) (2024)

## Built on by (most cited first, 30 of 56)

- [POSTA: A Go-to Framework for Customized Artistic Poster Generation](https://mlsanity.com/paper/2503.14908/) (2025)
- [From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective](https://mlsanity.com/paper/2503.13165/) (2025)
- [Quantum Variational Activation Functions Empower Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2509.14026/) (2025)
- [MedKAN: An Advanced Kolmogorov-Arnold Network for Medical Image Classification](https://mlsanity.com/paper/2502.18416/) (2025)
- [From Zero to Detail: A Progressive Spectral Decoupling Paradigm for UHD Image Restoration with New Benchmark](https://mlsanity.com/paper/2604.15654/) (2026)
- [Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability](https://mlsanity.com/paper/2501.09283/) (2025)
- [CVKAN: Complex-Valued Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2502.02417/) (2025)
- [Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention](https://mlsanity.com/paper/2502.13693/) (2025)
- [Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning](https://mlsanity.com/paper/2502.21130/) (2025)
- [Exploring Adversarial Transferability between Kolmogorov-arnold Networks](https://mlsanity.com/paper/2503.06276/) (2025)
- [Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning](https://mlsanity.com/paper/2509.13240/) (2025)
- [P1-KAN: an effective Kolmogorov-Arnold network with application to hydraulic valley optimization](https://mlsanity.com/paper/2410.03801/) (2024)
- [AF-KAN: Activation Function-Based Kolmogorov-Arnold Networks for Efficient Representation Learning](https://mlsanity.com/paper/2503.06112/) (2025)
- [KAN or MLP? Point Cloud Shows the Way Forward](https://mlsanity.com/paper/2504.13593/) (2025)
- [FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformer](https://mlsanity.com/paper/2505.13813/) (2025)
- [Improving Memory Efficiency for Training KANs via Meta Learning](https://mlsanity.com/paper/2506.07549/) (2025)
- [Automatic Grid Updates for Kolmogorov-Arnold Networks using Layer Histograms](https://mlsanity.com/paper/2511.08570/) (2025)
- [TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications](https://mlsanity.com/paper/2603.00482/) (2026)
- [Generative Quantum-inspired Kolmogorov-Arnold Eigensolver](https://mlsanity.com/paper/2605.04604/) (2026)
- [From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodology](https://mlsanity.com/paper/2412.17778/) (2024)
- [KAE: Kolmogorov-Arnold Auto-Encoder for Representation Learning](https://mlsanity.com/paper/2501.00420/) (2024)
- [Geometric Kolmogorov-Arnold Superposition Theorem](https://mlsanity.com/paper/2502.16664/) (2025)
- [Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?](https://mlsanity.com/paper/2503.10632/) (2025)
- [KANITE: Kolmogorov-Arnold Networks for ITE estimation](https://mlsanity.com/paper/2503.13912/) (2025)
- [Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models](https://mlsanity.com/paper/2506.14153/) (2025)
- [MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network](https://mlsanity.com/paper/2510.19105/) (2025)
- [SFFR: Spatial-Frequency Feature Reconstruction for Multispectral Aerial Object Detection](https://mlsanity.com/paper/2511.06298/) (2025)
- [SechKAN: Kolmogorov-Arnold Networks with Hyperbolic Secant Functions](https://mlsanity.com/paper/2607.18290/) (2026)
- [Early Prediction of Natural Gas Pipeline Leaks Using the MKTCN Model](https://mlsanity.com/paper/2411.06214/) (2024)
- [Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel NeRA Adapter for Enhanced Feature Adaptation](https://mlsanity.com/paper/2411.13901/) (2024)

## Abstract (arXiv)

Transformers stand as the cornerstone of mordern deep learning. Traditionally, these models rely on multi-layer perceptron (MLP) layers to mix the information between channels. In this paper, we introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces MLP layers with Kolmogorov-Arnold Network (KAN) layers to enhance the expressiveness and performance of the model. Integrating KANs into transformers, however, is no easy feat, especially when scaled up. Specifically, we identify three key challenges: (C1) Base function. The standard B-spline function used in KANs is not optimized for parallel computing on modern hardware, resulting in slower inference speeds. (C2) Parameter and Computation Inefficiency. KAN requires a unique function for each input-output pair, making the computation extremely large. (C3) Weight initialization. The initialization of weights in KANs is particularly challenging due to their learnable activation functions, which are critical for achieving convergence in deep neural networks. To overcome the aforementioned challenges, we propose three key solutions: (S1) Rational basis. We replace B-spline functions with rational functions to improve compatibility with modern GPUs. By implementing this in CUDA, we achieve faster computations. (S2) Group KAN. We share the activation weights through a group of neurons, to reduce the computational load without sacrificing performance. (S3) Variance-preserving initialization. We carefully initialize the activation weights to make sure that the activation variance is maintained across layers. With these designs, KAT scales effectively and readily outperforms traditional MLP-based transformers.