mlsanityData to 14 Oct 2026

FFT-based Dynamic Token Mixer for Vision

arXiv 2303.03932 ↗First posted 7 Mar 2023cs.CV cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Multi-head-self-attention (MHSA)-equipped models have achieved notable performance in computer vision. Their computational complexity is proportional to quadratic numbers of pixels in input feature maps, resulting in slow processing, especially when dealing with high-resolution images. New types of token-mixer are proposed as an alternative to MHSA to circumvent this problem: an FFT-based token-mixer involves global operations similar to MHSA but with lower computational complexity. However, despite its attractive properties, the FFT-based token-mixer has not been carefully examined in terms of its compatibility with the rapidly evolving MetaFormer architecture. Here, we propose a novel token-mixer called Dynamic Filter and novel image recognition models, DFFormer and CDFFormer, to close the gaps above. The results of image classification and downstream tasks, analysis, and visualization show that our models are helpful. Notably, their throughput and memory efficiency when dealing with high-resolution image recognition is remarkable. Our results indicate that Dynamic Filter is one of the token-mixer options that should be seriously considered. The code is available at https://github.com/okojoalg/dfformer

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 59 of its 75 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

59 of 75 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2016Layer Normalizationcited by 5,669
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2022A ConvNet for the 2020scited by 3,579
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2020Point Transformercited by 1,470
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  9. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  10. 2017Random Erasing Data Augmentationcited by 1,239
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2016Deep Networks with Stochastic Depthcited by 979
  13. 2020Designing Network Design Spacescited by 960
  14. 2021Going deeper with Image Transformerscited by 764
  15. 2020PCT: Point cloud transformercited by 732
  16. 2019Panoptic Feature Pyramid Networkscited by 645
  17. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  18. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  19. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  20. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  22. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  23. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 356
  24. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  25. 2016Dynamic Filter Networkscited by 319
  26. 2022How Do Vision Transformers Work?cited by 307
  27. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  28. 2021Video Transformer Networkcited by 272
  29. 2021Global Filter Networks for Image Classificationcited by 260
  30. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 248
  31. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  32. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  33. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  34. 2022Vision GNN: An Image is Worth Graph of Nodescited by 186
  35. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  36. 2022FlexiViT: One Model for All Patch Sizescited by 152
  37. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  38. 2022MetaFormer Baselines for Visioncited by 132

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 10; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 19
  2. 2025Frequency-Dynamic Attention Modulation for Dense Predictioncited by 10
  3. 2025FSDENet: A Frequency and Spatial Domains based Detail Enhancement Network for Remote Sensing Semantic Segmentationcited by 7
  4. 2024Complementary Advantages: Exploiting Cross-Field Frequency Correlation for NIR-Assisted Image Denoisingcited by 5
  5. 2025FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesiscited by 3
  6. 2025FourierSR: A Fourier Token-based Plugin for Efficient Image Super-Resolutioncited by 3
  7. 2025Dynamic Frequency Feature Fusion Network for Multi-Source Remote Sensing Data Classificationcited by 3
  8. 2024Signal-SGN: A Spiking Graph Convolutional Network for Skeletal Action Recognition via Learning Temporal-Frequency Dynamicscited by 2
  9. 2024Deep Fourier-embedded Network for RGB and Thermal Salient Object Detectioncited by 2
  10. 2024Efficient Fourier Filtering Network with Contrastive Learning for AAV-based Unaligned Bimodal Salient Object Detectioncited by 1
  11. 2024Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptationcited by 1
  12. 2025DAGNet: A Dual-View Attention-Guided Network for Efficient X-ray Security Inspectioncited by 1
10 more
  1. 2025Bridging the Inter-Domain Gap through Low-Level Features for Cross-Modal Medical Image Segmentationcited by 1
  2. 2026FSGNet: A Frequency-Aware and Semantic Guidance Network for Infrared Small Target Detectioncited by 1
  3. 2026CP-WSP: A Declarative CP-SAT Framework for Configurable Multi-Constraint Workforce Schedulingcited by 1
  4. 2025Spectral-Adaptive Modulation Networks for Visual Perceptioncited by 0
  5. 2025Forgery Guided Learning Strategy with Dual Perception Network for Deepfake Cross-domain Detectioncited by 0
  6. 2025Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imagingcited by 0
  7. 2026Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentationcited by 0
  8. 2026FreqTrack: Frequency Learning based Vision Transformer for RGB-Event Object Trackingcited by 0
  9. 2026Improving Radio Interferometry Imaging by Explicitly Modeling Cross-Domain Consistency in Reconstructioncited by 0
  10. 2026Denoising the Future: Context-Aware Spectral Diffusion for Temporal Knowledge Graph Extrapolationcited by 0