mlsanityData to 7 Oct 2026

AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder

arXiv 2306.06370 ↗First posted 10 Jun 2023cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

The recently introduced Segment Anything Model (SAM) combines a clever architecture and large quantities of training data to obtain remarkable image segmentation capabilities. However, it fails to reproduce such results for Out-Of-Distribution (OOD) domains such as medical images. Moreover, while SAM is conditioned on either a mask or a set of points, it may be desirable to have a fully automatic solution. In this work, we replace SAM's conditioning with an encoder that operates on the same input image. By adding this encoder and without further fine-tuning SAM, we obtain state-of-the-art results on multiple medical images and video benchmarks. This new encoder is trained via gradients provided by a frozen SAM. For inspecting the knowledge within it, and providing a lightweight segmentation solution, we also learn to decode it into a mask by a shallow deconvolution network.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

39 of 59 references.

References · 39

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  9. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2021GPT Understands, Toocited by 715
27 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  2. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
  3. 2019UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentationcited by 583
  4. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  5. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  6. 2020PraNet: Parallel Reverse Attention Network for Polyp Segmentationcited by 460
  7. 2018Enhanced-alignment Measure for Binary Foreground Map Evaluationcited by 370
  8. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  9. 2021Medical Transformer: Gated Axial-Attention for Medical Image Segmentationcited by 319
  10. 2016Gland Segmentation in Colon Histology Images: The GlaS Challenge Contestcited by 203
  11. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  12. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  13. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  14. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  15. 2015Salient Object Detection: A Benchmarkcited by 130
  16. 2021Shallow Attention Network for Polyp Segmentationcited by 107
  17. 2022Video Polyp Segmentation: A Deep Learning Perspectivecited by 81
  18. 2021Full-Duplex Strategy for Video Object Segmentationcited by 78
  19. 2019HarDNet: A Low Memory Traffic Networkcited by 76
  20. 2021Progressively Normalized Self-Attention Network for Video Polyp Segmentationcited by 75
  21. 2021HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPScited by 71
  22. 2020Motion-Attentive Transition for Zero-Shot Video Object Segmentationcited by 62
  23. 2023Adaptive Context Selection for Polyp Segmentationcited by 47
  24. 2021The Emergence of Objectness: Learning Zero-Shot Segmentation from Videoscited by 45
  25. 2021Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentationcited by 33
  26. 2022Zero-Shot Video Captioning with Evolving Pseudo-Tokenscited by 25
  27. 2021Duplex Contextual Relation Network for Polyp Segmentationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 31; 2025: 13; 2026: 16.

70 papers.

Sort:

By area:

  1. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  2. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  3. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  6. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 37
  7. 2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
  8. 2024Foundation Models for Biomedical Image Segmentation: A Surveycited by 28
  9. 2023AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentationcited by 19
  10. 2024MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAMcited by 17
  11. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  12. 2023nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performancecited by 12
38 more
  1. 2024ProtoSAM: One-Shot Medical Image Segmentation With Foundational Modelscited by 12
  2. 2025Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentationcited by 12
  3. 2024S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentationcited by 11
  4. 2024Generalizable Disaster Damage Assessment via Change Detection with Vision Foundation Modelcited by 10
  5. 2023Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectificationcited by 8
  6. 2025Recent Advances in Medical Imaging Segmentation: A Surveycited by 8
  7. 2024Sam2Rad: A Segmentation Model for Medical Images with Learnable Promptscited by 6
  8. 2024Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoningcited by 6
  9. 2023AdapterShadow: Adapting Segment Anything Model for Shadow Detectioncited by 5
  10. 2024SAMCT: Segment Any CT Allowing Labor-Free Task-Indicator Promptscited by 5
  11. 2024Automating MedSAM by Learning Prompts with Weak Few-Shot Supervisioncited by 5
  12. 2025TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasoundcited by 5
  13. 2023Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaulcited by 4
  14. 2023Exploring SAM Ablations for Enhancing Medical Segmentation in Radiology and Pathologycited by 4
  15. 2024MaskSAM: Towards Auto-prompt SAM with Mask Classification for Volumetric Medical Image Segmentationcited by 4
  16. 2024Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Modelscited by 4
  17. 2024Ultrasound SAM Adapter: Adapting SAM for Breast Lesion Segmentation in Ultrasound Imagescited by 4
  18. 2024EchoONE: Segmenting Multiple echocardiography Planes in One Modelcited by 4
  19. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentationcited by 4
  20. 2024ProMISe: Promptable Medical Image Segmentation using SAMcited by 3
  21. 2024PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentationcited by 3
  22. 2024AM-SAM: Automated Prompting and Mask Calibration for Segment Anything Modelcited by 3
  23. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  24. 2023AI-SAM: Automatic and Interactive Segment Anything Modelcited by 2
  25. 2024Foundation AI Model for Medical Image Segmentationcited by 2
  26. 2025Prompt learning with bounding box constraints for medical image segmentationcited by 2
  27. 2023Evaluation and improvement of Segment Anything Model for interactive histopathology image segmentationcited by 1
  28. 2024Leveraging SAM for Single-Source Domain Generalization in Medical Image Segmentationcited by 1
  29. 2024PolypNextLSTM: A lightweight and fast polyp video segmentation network using ConvNext and ConvLSTMcited by 1
  30. 2024Segment Any Medical Model Extendedcited by 1
  31. 2024Lung-CADex: Fully automatic Zero-Shot Detection and Classification of Lung Nodules in Thoracic CT Imagescited by 1
  32. 2024Urban Waterlogging Detection: A Challenging Benchmark and Large-Small Model Co-Adaptercited by 1
  33. 2024Foundation Models for Amodal Video Instance Segmentation in Automated Drivingcited by 1
  34. 2024A SAM-guided and Match-based Semi-Supervised Segmentation Framework for Medical Imagingcited by 1
  35. 2025Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentationcited by 1
  36. 2025SAM2-SGP: Enhancing SAM2 for Medical Image Segmentation via Support-Set Guided Promptingcited by 1
  37. 2025Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learningcited by 1
  38. 2025LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-trainingcited by 1

50 most cited of 70 All 70 (JSON)