# FourierKAN outperforms MLP on Text Classification Head Fine-tuning

arXiv: https://arxiv.org/abs/2408.08803 · HTML: https://mlsanity.com/paper/2408.08803/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Kolmogorov-Arnold Networks are Radial Basis Function Networks](https://mlsanity.com/paper/2405.06721/) (2024)
- [Wav-KAN: Wavelet Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2405.12832/) (2024)
- [Kolmogorov-Arnold Networks (KANs) for Time Series Analysis](https://mlsanity.com/paper/2405.08790/) (2024)
- [DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems](https://mlsanity.com/paper/2405.19143/) (2024)
- [Kolmogorov-Arnold Networks for Time Series: Bridging Predictive Power and Interpretability](https://mlsanity.com/paper/2406.02496/) (2024)
- [fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions](https://mlsanity.com/paper/2406.07456/) (2024)
- [A Temporal Kolmogorov-Arnold Transformer for Time Series Forecasting](https://mlsanity.com/paper/2406.02486/) (2024)
- [Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensing](https://mlsanity.com/paper/2406.00600/) (2024)
- [BSRBF-KAN: A combination of B-splines and Radial Basis Functions in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2406.11173/) (2024)
- [Enhancing Graph Collaborative Filtering with FourierKAN Feature Transformation](https://mlsanity.com/paper/2406.01034/) (2024)
- [KANQAS: Kolmogorov-Arnold Network for Quantum Architecture Search](https://mlsanity.com/paper/2406.17630/) (2024)
- [Smooth Kolmogorov Arnold networks enabling structural knowledge representation](https://mlsanity.com/paper/2405.11318/) (2024)
- [Parameter-Efficient Tuning Makes a Good Classification Head](https://mlsanity.com/paper/2210.16771/) (2022)
- [Tuning Pre-trained Model via Moment Probing](https://mlsanity.com/paper/2307.11342/) (2023)

## Built on by (most cited first, 5 of 5)

- [KANEL'E: Kolmogorov-Arnold Networks for Efficient LUT-based Evaluation](https://mlsanity.com/paper/2512.12850/) (2025)
- [SPPSFormer: High-quality Superpoint-based Transformer for Roof Plane Instance Segmentation from Point Clouds](https://mlsanity.com/paper/2505.24475/) (2025)
- [Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling](https://mlsanity.com/paper/2509.00560/) (2025)
- [Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning](https://mlsanity.com/paper/2511.21081/) (2025)
- [IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting](https://mlsanity.com/paper/2603.24207/) (2026)

## Abstract (arXiv)

In resource constraint settings, adaptation to downstream classification tasks involves fine-tuning the final layer of a classifier (i.e. classification head) while keeping rest of the model weights frozen. Multi-Layer Perceptron (MLP) heads fine-tuned with pre-trained transformer backbones have long been the de facto standard for text classification head fine-tuning. However, the fixed non-linearity of MLPs often struggles to fully capture the nuances of contextual embeddings produced by pre-trained models, while also being computationally expensive. In our work, we investigate the efficacy of KAN and its variant, Fourier KAN (FR-KAN), as alternative text classification heads. Our experiments reveal that FR-KAN significantly outperforms MLPs with an average improvement of 10% in accuracy and 11% in F1-score across seven pre-trained transformer models and four text classification tasks. Beyond performance gains, FR-KAN is more computationally efficient and trains faster with fewer parameters. These results underscore the potential of FR-KAN to serve as a lightweight classification head, with broader implications for advancing other Natural Language Processing (NLP) tasks.