# Adaptive Frequency Filters As Efficient Global Token Mixers

arXiv: https://arxiv.org/abs/2307.14008 · HTML: https://mlsanity.com/paper/2307.14008/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 90 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 30 of 69)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)

## Built on by (most cited first, 30 of 45)

- [Frequency-aware Feature Fusion for Dense Image Prediction](https://mlsanity.com/paper/2408.12879/) (2024)
- [Frequency-Adaptive Dilated Convolution for Semantic Segmentation](https://mlsanity.com/paper/2403.05369/) (2024)
- [DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space](https://mlsanity.com/paper/2412.15032/) (2024)
- [Frequency Dynamic Convolution for Dense Image Prediction](https://mlsanity.com/paper/2503.18783/) (2025)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [Neural Spectral Decomposition for Dataset Distillation](https://mlsanity.com/paper/2408.16236/) (2024)
- [Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approach](https://mlsanity.com/paper/2310.12004/) (2023)
- [When Semantic Segmentation Meets Frequency Aliasing](https://mlsanity.com/paper/2403.09065/) (2024)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)
- [MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization](https://mlsanity.com/paper/2406.01460/) (2024)
- [SPIRONet: Spatial-Frequency Learning and Graph-based Channel Interaction Network for Vessel Segmentation](https://mlsanity.com/paper/2406.19749/) (2024)
- [Deep Tree Tensor Networks](https://mlsanity.com/paper/2502.09928/) (2025)
- [Dual-Perspective United Transformer for Object Segmentation in Optical Remote Sensing Images](https://mlsanity.com/paper/2506.21866/) (2025)
- [Butter: Frequency Consistency and Hierarchical Fusion for Autonomous Driving Object Detection](https://mlsanity.com/paper/2507.13373/) (2025)
- [Vision Transformers are Circulant Attention Learners](https://mlsanity.com/paper/2512.21542/) (2025)
- [Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis](https://mlsanity.com/paper/2410.04703/) (2024)
- [FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis](https://mlsanity.com/paper/2503.04067/) (2025)
- [FourierSR: A Fourier Token-based Plugin for Efficient Image Super-Resolution](https://mlsanity.com/paper/2503.10043/) (2025)
- [Spatial Frequency Modulation for Semantic Segmentation](https://mlsanity.com/paper/2507.11893/) (2025)
- [SpiralMLP: A Lightweight Vision MLP Architecture](https://mlsanity.com/paper/2404.00648/) (2024)
- [Self-supervised Gait-based Emotion Representation Learning from Selective Strongly Augmented Skeleton Sequences](https://mlsanity.com/paper/2405.04900/) (2024)
- [AFIDAF: Alternating Fourier and Image Domain Adaptive Filters as an Efficient Alternative to Attention in ViTs](https://mlsanity.com/paper/2407.12217/) (2024)
- [Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification](https://mlsanity.com/paper/2412.13232/) (2024)
- [UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations](https://mlsanity.com/paper/2510.24116/) (2025)
- [FSGNet: A Frequency-Aware and Semantic Guidance Network for Infrared Small Target Detection](https://mlsanity.com/paper/2603.25389/) (2026)
- [CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Scheduling](https://mlsanity.com/paper/2607.05177/) (2026)
- [Mixer is more than just a model](https://mlsanity.com/paper/2402.18007/) (2024)
- [DFE-IANet: A Method for Polyp Image Classification Based on Dual-domain Feature Extraction and Interaction Attention](https://mlsanity.com/paper/2407.20843/) (2024)
- [NukesFormers: Unpaired Hyperspectral Image Generation with Non-Uniform Domain Alignment](https://mlsanity.com/paper/2503.07004/) (2025)
- [Adaptive Wavelet Filters as Practical Texture Feature Amplifiers for Parkinson's Disease Screening in OCT](https://mlsanity.com/paper/2503.19292/) (2025)

## Abstract (arXiv)

Recent vision transformers, large-kernel CNNs and MLPs have attained remarkable successes in broad vision tasks thanks to their effective information fusion in the global scope. However, their efficient deployments, especially on mobile devices, still suffer from noteworthy challenges due to the heavy computational costs of self-attention mechanisms, large kernels, or fully connected layers. In this work, we apply conventional convolution theorem to deep learning for addressing this and reveal that adaptive frequency filters can serve as efficient global token mixers. With this insight, we propose Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers a latent representation to the frequency domain via a Fourier transform and performs semantic-adaptive frequency filtering via an elementwise multiplication, which mathematically equals to a token mixing operation in the original latent space with a dynamic convolution kernel as large as the spatial resolution of this latent representation. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet. Extensive experiments demonstrate the effectiveness of our proposed AFF token mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs compared to other lightweight network designs on broad visual tasks, including visual recognition and dense prediction tasks.