# Visual Fourier Prompt Tuning

arXiv: https://arxiv.org/abs/2411.01327 · HTML: https://mlsanity.com/paper/2411.01327/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 109 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 30 of 71)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)

## Built on by (most cited first, 30 of 30)

- [Visual Instance-aware Prompt Tuning](https://mlsanity.com/paper/2507.07796/) (2025)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [Visual Variational Autoencoder Prompt Tuning](https://mlsanity.com/paper/2503.17650/) (2025)
- [Re-Imagining Multimodal Instruction Tuning: A Representation View](https://mlsanity.com/paper/2503.00723/) (2025)
- [Doctor Approved: Generating Medically Accurate Skin Disease Images through AI-Expert Feedback](https://mlsanity.com/paper/2506.12323/) (2025)
- [All You Need is One: Capsule Prompt Tuning with a Single Vector](https://mlsanity.com/paper/2510.16670/) (2025)
- [Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts](https://mlsanity.com/paper/2501.18936/) (2025)
- [WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection](https://mlsanity.com/paper/2510.05305/) (2025)
- [Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision](https://mlsanity.com/paper/2504.21423/) (2025)
- [Token Coordinated Prompt Attention is Needed for Visual Prompting](https://mlsanity.com/paper/2505.02406/) (2025)
- [Vision Graph Prompting via Semantic Low-Rank Decomposition](https://mlsanity.com/paper/2505.04121/) (2025)
- [Probabilistic Token Alignment for Large Language Model Fusion](https://mlsanity.com/paper/2509.17276/) (2025)
- [State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding](https://mlsanity.com/paper/2510.12160/) (2025)
- [Visual Prompt-Agnostic Evolution](https://mlsanity.com/paper/2601.20232/) (2026)
- [Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding](https://mlsanity.com/paper/2602.18977/) (2026)
- [Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models](https://mlsanity.com/paper/2606.26379/) (2026)
- [Structured Relational Reasoning for Group Activity Assessment](https://mlsanity.com/paper/2508.07996/) (2025)
- [VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion](https://mlsanity.com/paper/2510.16446/) (2025)
- [GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models](https://mlsanity.com/paper/2506.11042/) (2025)
- [Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation](https://mlsanity.com/paper/2509.19733/) (2025)
- [Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation](https://mlsanity.com/paper/2510.07823/) (2025)
- [CASP: Few-Shot Class-Incremental Learning with CLS Token Attention Steering Prompts](https://mlsanity.com/paper/2601.16773/) (2026)
- [TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching](https://mlsanity.com/paper/2601.19739/) (2026)
- [Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters](https://mlsanity.com/paper/2603.18101/) (2026)
- [Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression](https://mlsanity.com/paper/2603.27383/) (2026)
- [Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso](https://mlsanity.com/paper/2604.03953/) (2026)
- [Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?](https://mlsanity.com/paper/2606.03837/) (2026)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)
- [Adapting Vision Foundation Models with Cascaded Semantics](https://mlsanity.com/paper/2608.05393/) (2026)
- [GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model](https://mlsanity.com/paper/2609.19716/) (2026)

## Abstract (arXiv)

With the scale of vision Transformer-based models continuing to grow, finetuning these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as a parameter-efficient finetuning (PEFT) method to this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there is a substantial disparity between the datasets applied in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Apart from its inherent simplicity and intuitiveness, VFPT exhibits superior performance across all datasets, offering a general solution to dataset challenges, irrespective of data disparities. Empirical results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks, with low parameter usage (e.g., 0.57% of model parameters on VTAB-1k) and notable performance enhancements (e.g., 73.20% of mean accuracy on VTAB-1k). Our code is avaliable at https://github.com/runtsang/VFPT.