# SpectFormer: Frequency and Attention is what you need in a Vision Transformer

arXiv: https://arxiv.org/abs/2304.06446 · HTML: https://mlsanity.com/paper/2304.06446/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 90 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)

## Built on by (most cited first, 30 of 48)

- [SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series](https://mlsanity.com/paper/2403.15360/) (2024)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [P-Mamba: Marrying Perona Malik Diffusion with Mamba for Efficient Pediatric Echocardiographic Left Ventricular Segmentation](https://mlsanity.com/paper/2402.08506/) (2024)
- [Scattering Vision Transformer: Spectral Mixing Matters](https://mlsanity.com/paper/2311.01310/) (2023)
- [LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition](https://mlsanity.com/paper/2402.00033/) (2024)
- [Graph Convolutions Enrich the Self-Attention in Transformers!](https://mlsanity.com/paper/2312.04234/) (2023)
- [Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2312.12754/) (2023)
- [Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation](https://mlsanity.com/paper/2401.16923/) (2024)
- [FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networks](https://mlsanity.com/paper/2501.14744/) (2024)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)
- [Surya: Foundation Model for Heliophysics](https://mlsanity.com/paper/2508.14112/) (2025)
- [Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis](https://mlsanity.com/paper/2403.18063/) (2024)
- [Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights](https://mlsanity.com/paper/2403.19882/) (2024)
- [Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain](https://mlsanity.com/paper/2505.08516/) (2025)
- [AI Foundation Model for Heliophysics: Applications, Design, and Implementation](https://mlsanity.com/paper/2410.10841/) (2024)
- [Scalable Event Cloud Network for Event-based Classification](https://mlsanity.com/paper/2412.20803/) (2024)
- [HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet](https://mlsanity.com/paper/2604.14724/) (2026)
- [Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers](https://mlsanity.com/paper/2405.13901/) (2024)
- [Signal-SGN: A Spiking Graph Convolutional Network for Skeletal Action Recognition via Learning Temporal-Frequency Dynamics](https://mlsanity.com/paper/2408.01701/) (2024)
- [EHCTNet: Enhanced Hybrid of CNN and Transformer Network for Remote Sensing Image Change Detection](https://mlsanity.com/paper/2501.01238/) (2025)
- [D3R-DETR: DETR with Dual-Domain Density Refinement for Tiny Object Detection in Aerial Images](https://mlsanity.com/paper/2601.02747/) (2026)
- [Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding](https://mlsanity.com/paper/2602.18977/) (2026)
- [DEFormer: DCT-driven Enhancement Transformer for Low-light Image and Dark Vision](https://mlsanity.com/paper/2309.06941/) (2023)
- [AFIDAF: Alternating Fourier and Image Domain Adaptive Filters as an Efficient Alternative to Attention in ViTs](https://mlsanity.com/paper/2407.12217/) (2024)
- [MxT: Mamba x Transformer for Image Inpainting](https://mlsanity.com/paper/2407.16126/) (2024)
- [SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit](https://mlsanity.com/paper/2503.00060/) (2025)
- [Auto-US: An Ultrasound Video Diagnosis Agent Using Video Classification Framework and LLMs](https://mlsanity.com/paper/2511.07748/) (2025)
- [TimeAPN: Adaptive Amplitude-Phase Non-Stationarity Normalization for Time Series Forecasting](https://mlsanity.com/paper/2603.17436/) (2026)
- [Efficiency Follows Global-Local Decoupling](https://mlsanity.com/paper/2603.19567/) (2026)
- [NAKUL-Med: Spectral-Graph State Space Models with Dynamics Kernels for Medical Signals](https://mlsanity.com/paper/2605.00871/) (2026)

## Abstract (arXiv)

Vision transformers have been applied successfully for image recognition tasks. There have been either multi-headed self-attention based (ViT citedosovitskiy2020image, DeIT, citetouvron2021training) similar to the original work in textual models or more recently based on spectral layers (Fnetcitelee2021fnet, GFNetciterao2021global, AFNOciteguibas2021efficient). We hypothesize that both spectral and multi-headed attention plays a major role. We investigate this hypothesis through this work and observe that indeed combining spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance, it improves the top-1 accuracy by 2% on ImageNet compared to both GFNet-H and LiT. SpectFormer-S reaches 84.25% top-1 accuracy on ImageNet-1K (state of the art for small version). Further, Spectformer-L achieves 85.7% that is the state of the art for the comparable base version of the transformers. We further ensure that we obtain reasonable results in other scenarios such as transfer learning on standard datasets such as CIFAR-10, CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate its use in downstream tasks such of object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. Hence, we believe that combined spectral and attention layers are what are needed for vision transformers.