mlsanityData to 14 Oct 2026

FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization

arXiv 2303.14189 ↗First posted 24 Mar 2023cs.CVcited by 162

Not on the map. Why?

Abstract (arXiv)

The recent amalgamation of transformer and convolutional designs has led to steady improvements in accuracy and efficiency of the models. In this work, we introduce FastViT, a hybrid vision transformer architecture that obtains the state-of-the-art latency-accuracy trade-off. To this end, we introduce a novel token mixing operator, RepMixer, a building block of FastViT, that uses structural reparameterization to lower the memory access cost by removing skip-connections in the network. We further apply train-time overparametrization and large kernel convolutions to boost accuracy and empirically show that these choices have minimal effect on latency. We show that - our model is 3.5x faster than CMT, a recent state-of-the-art hybrid transformer architecture, 4.9x faster than EfficientNet, and 1.9x faster than ConvNeXt on a mobile device for the same accuracy on the ImageNet dataset. At similar latency, our model obtains 4.2% better Top-1 accuracy on ImageNet than MobileOne. Our model consistently outperforms competing architectures across several tasks -- image classification, detection, segmentation and 3D mesh regression with significant improvement in latency on both a mobile device and a desktop GPU. Furthermore, our model is highly robust to out-of-distribution samples and corruptions, improving over competing robust models. Code and models are available at https://github.com/apple/ml-fastvit.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Builds on 69 of its 79 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

69 of 79 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2017Mask R-CNNcited by 7,277
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  12. 2022A ConvNet for the 2020scited by 3,579
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  3. 2019Searching for MobileNetV3cited by 1,994
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  8. 2019Natural Adversarial Examplescited by 1,408
  9. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  11. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  13. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  15. 2020Designing Network Design Spacescited by 960
  16. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  17. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  18. 2021Going deeper with Image Transformerscited by 764
  19. 2019Panoptic Feature Pyramid Networkscited by 645
  20. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  21. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  22. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  23. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  24. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  25. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  26. 2020ResNeSt: Split-Attention Networkscited by 445
  27. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  28. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  29. 2021Early Convolutions Help Transformers See Bettercited by 396
  30. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  31. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  32. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  33. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  34. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  35. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  36. 2021Mesh Graphormercited by 265
  37. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  38. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 55; 2025: 46; 2026: 42.

162 papers.

Sort:

By area:

  1. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 124
  2. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  3. 2024DiTFastAttn: Attention Compression for Diffusion Transformer Modelscited by 115
  4. 2024MobileNetV4 -- Universal Models for the Mobile Ecosystemcited by 99
  5. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  6. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  7. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  8. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 56
  9. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  10. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  11. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 34
  12. 2024A Simple Baseline for Efficient Hand Mesh Reconstructioncited by 27
38 more
  1. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  2. 2024GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNscited by 21
  3. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  4. 2024SFFNet: A Wavelet-Based Spatial and Frequency Domain Fusion Network for Remote Sensing Segmentationcited by 17
  5. 2024EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Dualitycited by 17
  6. 2025FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attentioncited by 17
  7. 2024MobileMamba: Lightweight Multi-Receptive Visual Mamba Networkcited by 16
  8. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 15
  9. 2024TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mambacited by 15
  10. 2025LSNet: See Large, Focus Smallcited by 15
  11. 2024WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radarcited by 13
  12. 2024Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Streamcited by 13
  13. 2024HydraViT: Stacking Heads for a Scalable ViTcited by 11
  14. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  15. 2023RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNetscited by 9
  16. 2023SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computerscited by 9
  17. 2024Scaling Graph Convolutions for Mobile Visioncited by 9
  18. 2024HandOS: 3D Hand Reconstruction in One Stagecited by 9
  19. 2025iFormer: Integrating ConvNet and Transformer for Mobile Applicationcited by 9
  20. 2025Context-guided Responsible Data Augmentation with Diffusion Modelscited by 9
  21. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  22. 2024Cross-video Identity Correlating for Person Re-identification Pre-trainingcited by 8
  23. 2024Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representationscited by 8
  24. 2025MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Devicecited by 8
  25. 2023DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Modelscited by 7
  26. 2024CLIP with Quality Captions: A Strong Pretraining for Vision Taskscited by 7
  27. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  28. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  29. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
  30. 2024HSViT: Horizontally Scalable Vision Transformercited by 6
  31. 2024LowFormer: Hardware Efficient Design for Convolutional Transformer Backbonescited by 6
  32. 2025ClusterViG: Efficient Globally Aware Vision GNNs via Image Partitioningcited by 6
  33. 2025Learning to Drive from a World Modelcited by 6
  34. 2023Semantic Segmentation in Satellite Hyperspectral Imagery by Deep Learningcited by 5
  35. 2024Lite-SAM Is Actually What You Need for Segment Everythingcited by 5
  36. 2024PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformercited by 5
  37. 2024Residual Kolmogorov-Arnold Network for Enhanced Deep Learningcited by 5
  38. 2024CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interactioncited by 5

50 most cited of 162 All 162 (JSON)