# RepViT-SAM: Towards Real-Time Segmenting Anything

arXiv: https://arxiv.org/abs/2312.05760 · HTML: https://mlsanity.com/paper/2312.05760/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [Segment Any Anomaly without Training via Hybrid Prompt Regularization](https://mlsanity.com/paper/2305.10724/) (2023)
- [Make RepVGG Greater Again: A Quantization-aware Approach](https://mlsanity.com/paper/2212.01593/) (2022)

## Built on by (most cited first, 17 of 17)

- [Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2412.10137/) (2024)
- [EdgeSAM: Prompt-In-the-Loop Distillation for SAM](https://mlsanity.com/paper/2312.06660/) (2023)
- [EdgeTAM: On-Device Track Anything Model](https://mlsanity.com/paper/2501.07256/) (2025)
- [PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation](https://mlsanity.com/paper/2403.09192/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images](https://mlsanity.com/paper/2403.09827/) (2024)
- [De-LightSAM: Modality-Decoupled Lightweight SAM for Generalizable Medical Segmentation](https://mlsanity.com/paper/2407.14153/) (2024)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [UniUltra: Interactive Parameter-Efficient SAM2 for Universal Ultrasound Segmentation](https://mlsanity.com/paper/2511.15771/) (2025)
- [MCP-MedSAM: A Powerful Lightweight Medical Segment Anything Model Trained with a Single GPU in Just One Day](https://mlsanity.com/paper/2412.05888/) (2024)
- [Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework](https://mlsanity.com/paper/2506.02854/) (2025)
- [Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation](https://mlsanity.com/paper/2504.02351/) (2025)
- [SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning](https://mlsanity.com/paper/2606.08992/) (2026)
- [SAM3-LiteText: An Anatomical Study of the SAM3 Text Encoder for Efficient Vision-Language Segmentation](https://mlsanity.com/paper/2602.12173/) (2026)
- [OpenNav: Efficient Open Vocabulary 3D Object Detection for Smart Wheelchair Navigation](https://mlsanity.com/paper/2408.13936/) (2024)
- [SinkSAM-Net: Knowledge-Driven Self-Supervised Sinkhole Segmentation Using Topographic Priors and Segment Anything Model](https://mlsanity.com/paper/2410.01473/) (2024)
- [RRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations](https://mlsanity.com/paper/2601.01865/) (2026)

## Abstract (arXiv)

Segment Anything Model (SAM) has shown impressive zero-shot transfer performance for various computer vision tasks recently. However, its heavy computation costs remain daunting for practical applications. MobileSAM proposes to replace the heavyweight image encoder in SAM with TinyViT by employing distillation, which results in a significant reduction in computational requirements. However, its deployment on resource-constrained mobile devices still encounters challenges due to the substantial memory and computational overhead caused by self-attention mechanisms. Recently, RepViT achieves the state-of-the-art performance and latency trade-off on mobile devices by incorporating efficient architectural designs of ViTs into CNNs. Here, to achieve real-time segmenting anything on mobile devices, following MobileSAM, we replace the heavyweight image encoder in SAM with RepViT model, ending up with the RepViT-SAM model. Extensive experiments show that RepViT-SAM can enjoy significantly better zero-shot transfer capability than MobileSAM, along with nearly 10× faster inference speed. The code and models are available at urlhttps://github.com/THU-MIG/RepViT.