# FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization

arXiv: https://arxiv.org/abs/2303.14189 · HTML: https://mlsanity.com/paper/2303.14189/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 79 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 30 of 69)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)

## Built on by (most cited first, 30 of 162)

- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [DiTFastAttn: Attention Compression for Diffusion Transformer Models](https://mlsanity.com/paper/2406.08552/) (2024)
- [MobileNetV4 -- Universal Models for the Mobile Ecosystem](https://mlsanity.com/paper/2404.10518/) (2024)
- [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://mlsanity.com/paper/2311.17049/) (2023)
- [FastVLM: Efficient Vision Encoding for Vision Language Models](https://mlsanity.com/paper/2412.13303/) (2024)
- [UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition](https://mlsanity.com/paper/2311.15599/) (2023)
- [SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design](https://mlsanity.com/paper/2401.16456/) (2024)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation](https://mlsanity.com/paper/2601.22153/) (2026)
- [A Simple Baseline for Efficient Hand Mesh Reconstruction](https://mlsanity.com/paper/2403.01813/) (2024)
- [Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://mlsanity.com/paper/2310.04621/) (2023)
- [GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs](https://mlsanity.com/paper/2405.06849/) (2024)
- [MobileCLIP2: Improving Multi-Modal Reinforced Training](https://mlsanity.com/paper/2508.20691/) (2025)
- [SFFNet: A Wavelet-Based Spatial and Frequency Domain Fusion Network for Remote Sensing Segmentation](https://mlsanity.com/paper/2405.01992/) (2024)
- [EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality](https://mlsanity.com/paper/2411.15241/) (2024)
- [FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention](https://mlsanity.com/paper/2512.01540/) (2025)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)
- [Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models](https://mlsanity.com/paper/2311.18237/) (2023)
- [TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mamba](https://mlsanity.com/paper/2411.17473/) (2024)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radar](https://mlsanity.com/paper/2403.12686/) (2024)
- [Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream](https://mlsanity.com/paper/2411.05712/) (2024)
- [HydraViT: Stacking Heads for a Scalable ViT](https://mlsanity.com/paper/2409.17978/) (2024)
- [Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers](https://mlsanity.com/paper/2308.09372/) (2023)
- [RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets](https://mlsanity.com/paper/2310.10563/) (2023)
- [SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computers](https://mlsanity.com/paper/2311.03747/) (2023)
- [Scaling Graph Convolutions for Mobile Vision](https://mlsanity.com/paper/2406.05850/) (2024)
- [HandOS: 3D Hand Reconstruction in One Stage](https://mlsanity.com/paper/2412.01537/) (2024)

## Abstract (arXiv)

The recent amalgamation of transformer and convolutional designs has led to steady improvements in accuracy and efficiency of the models. In this work, we introduce FastViT, a hybrid vision transformer architecture that obtains the state-of-the-art latency-accuracy trade-off. To this end, we introduce a novel token mixing operator, RepMixer, a building block of FastViT, that uses structural reparameterization to lower the memory access cost by removing skip-connections in the network. We further apply train-time overparametrization and large kernel convolutions to boost accuracy and empirically show that these choices have minimal effect on latency. We show that - our model is 3.5x faster than CMT, a recent state-of-the-art hybrid transformer architecture, 4.9x faster than EfficientNet, and 1.9x faster than ConvNeXt on a mobile device for the same accuracy on the ImageNet dataset. At similar latency, our model obtains 4.2% better Top-1 accuracy on ImageNet than MobileOne. Our model consistently outperforms competing architectures across several tasks -- image classification, detection, segmentation and 3D mesh regression with significant improvement in latency on both a mobile device and a desktop GPU. Furthermore, our model is highly robust to out-of-distribution samples and corruptions, improving over competing robust models. Code and models are available at https://github.com/apple/ml-fastvit.