# SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation

arXiv: https://arxiv.org/abs/2408.08870 · HTML: https://mlsanity.com/paper/2408.08870/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 77 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 30 of 40)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles](https://mlsanity.com/paper/2306.00989/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Anabranch Network for Camouflaged Object Segmentation](https://mlsanity.com/paper/2105.09451/) (2021)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [Simultaneously Localize, Segment and Rank the Camouflaged Objects](https://mlsanity.com/paper/2103.04011/) (2021)
- [Visual Saliency Based on Multiscale Deep Features](https://mlsanity.com/paper/1503.08663/) (2015)
- [Camouflaged Object Segmentation with Distraction Mining](https://mlsanity.com/paper/2104.10475/) (2021)
- [Zoom In and Out: A Mixed-scale Triplet Network for Camouflaged Object Detection](https://mlsanity.com/paper/2203.02688/) (2022)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Receptive Field Block Net for Accurate and Fast Object Detection](https://mlsanity.com/paper/1711.07767/) (2017)
- [Salient Object Detection via Integrity Learning](https://mlsanity.com/paper/2101.07663/) (2021)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [Shallow Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2108.00882/) (2021)
- [Context-aware Cross-level Fusion Network for Camouflaged Object Detection](https://mlsanity.com/paper/2105.12555/) (2021)

## Built on by (most cited first, 30 of 64)

- [Efficient Track Anything](https://mlsanity.com/paper/2411.18933/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Dino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentation](https://mlsanity.com/paper/2508.20909/) (2025)
- [ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object](https://mlsanity.com/paper/2503.12006/) (2025)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation](https://mlsanity.com/paper/2409.18653/) (2024)
- [Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2](https://mlsanity.com/paper/2510.24195/) (2025)
- [NTIRE 2025 challenge on Text to Image Generation Model Quality Assessment](https://mlsanity.com/paper/2505.16314/) (2025)
- [SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost](https://mlsanity.com/paper/2506.01304/) (2025)
- [CamSAM2: Segment Anything Accurately in Camouflaged Videos](https://mlsanity.com/paper/2503.19730/) (2025)
- [TP-Seg: Task-Prototype Framework for Unified Medical Lesion Segmentation](https://mlsanity.com/paper/2604.00684/) (2026)
- [MedicoSAM: Robust Improvement of SAM for Medical Imaging](https://mlsanity.com/paper/2501.11734/) (2025)
- [STPNet: Scale-aware Text Prompt Network for Medical Image Segmentation](https://mlsanity.com/paper/2504.01561/) (2025)
- [DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance](https://mlsanity.com/paper/2606.08906/) (2026)
- [LoD-Loc v2: Aerial Visual Localization over Low Level-of-Detail City Models using Explicit Silhouette Alignment](https://mlsanity.com/paper/2507.00659/) (2025)
- [Evaluating SAM2 for Video Semantic Segmentation](https://mlsanity.com/paper/2512.01774/) (2025)
- [SAM3-UNet: Simplified Adaptation of Segment Anything Model 3](https://mlsanity.com/paper/2512.01789/) (2025)
- [SpaceSeg: A High-Precision Intelligent Perception Segmentation Method for Multi-Spacecraft On-Orbit Targets](https://mlsanity.com/paper/2503.11133/) (2025)
- [Synergistic Bleeding Region and Point Detection in Laparoscopic Surgical Videos](https://mlsanity.com/paper/2503.22174/) (2025)
- [Semantic-Guided Diffusion Model for Single-Step Image Super-Resolution](https://mlsanity.com/paper/2505.07071/) (2025)
- [When SAM2 Meets Video Shadow and Mirror Detection](https://mlsanity.com/paper/2412.19293/) (2024)
- [Kolmogorov-Arnold Network for Remote Sensing Image Semantic Segmentation](https://mlsanity.com/paper/2501.07390/) (2025)
- [Frequency Domain Enhanced U-Net for Low-Frequency Information-Rich Image Segmentation in Surgical and Deep-Sea Exploration Robots](https://mlsanity.com/paper/2502.03829/) (2025)
- [A Racing Dataset and Baseline Model for Track Detection in Autonomous Racing](https://mlsanity.com/paper/2502.14068/) (2025)
- [SAM2-ELNet: Label Enhancement and Automatic Annotation for Remote Sensing Segmentation](https://mlsanity.com/paper/2503.12404/) (2025)
- [DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement](https://mlsanity.com/paper/2503.21187/) (2025)
- [ChronoTailor: Harnessing Attention Guidance for Fine-Grained Video Virtual Try-On](https://mlsanity.com/paper/2506.05858/) (2025)
- [SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Tasks](https://mlsanity.com/paper/2508.03566/) (2025)
- [Privacy-enhancing Sclera Segmentation Benchmarking Competition: SSBC 2025](https://mlsanity.com/paper/2508.10737/) (2025)

## Abstract (arXiv)

Image segmentation plays an important role in vision understanding. Recently, the emerging vision foundation models continuously achieved superior performance on various tasks. Following such success, in this paper, we prove that the Segment Anything Model 2 (SAM2) can be a strong encoder for U-shaped segmentation models. We propose a simple but effective framework, termed SAM2-UNet, for versatile image segmentation. Specifically, SAM2-UNet adopts the Hiera backbone of SAM2 as the encoder, while the decoder uses the classic U-shaped design. Additionally, adapters are inserted into the encoder to allow parameter-efficient fine-tuning. Preliminary experiments on various downstream tasks, such as camouflaged object detection, salient object detection, marine animal segmentation, mirror detection, and polyp segmentation, demonstrate that our SAM2-UNet can simply beat existing specialized state-of-the-art methods without bells and whistles. Project page: urlhttps://github.com/WZH0120/SAM2-UNet.