mlsanityData to 7 Oct 2026

Personalize Segment Anything Model with One Shot

arXiv 2305.03048 ↗First posted 4 May 2023cs.CV cs.AI cs.CLcited by 250

Not on the map. Why?

Abstract (arXiv)

Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models. Despite the generality, customizing SAM for specific visual concepts without man-powered prompting is under explored, e.g., automatically segmenting your pet dog in different images. In this paper, we propose a training-free Personalization approach for SAM, termed as PerSAM. Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior, and segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement. In this way, we effectively adapt SAM for private use without any training. To further alleviate the mask ambiguity, we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce two learnable weights for multi-scale masks, only training 2 parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for personalized evaluation, and test our methods on video object segmentation with competitive performance. Besides, our approach can also enhance DreamBooth to personalize Stable Diffusion for text-to-image generation, which discards the background disturbance for better target appearance learning. Code is released at https://github.com/ZrrSkywalker/Personalize-SAM

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). Builds on 72 of its 89 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

72 of 89 references.

References · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Segment Anythingcited by 8,158
  10. 2017Mask R-CNNcited by 7,259
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  8. 2016Pyramid Scene Parsing Networkcited by 2,472
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2022Visual Prompt Tuningcited by 1,674
  14. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  15. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  17. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  18. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  19. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  20. 2018Panoptic Segmentationcited by 796
  21. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  23. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  24. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  25. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  26. 2017Learning multiple visual domains with residual adapterscited by 553
  27. 2023Segment Everything Everywhere All at Oncecited by 546
  28. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  29. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  30. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  31. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  32. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  33. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  34. 2020Conditional Convolutions for Instance Segmentationcited by 314
  35. 2022Visual Prompting via Image Inpaintingcited by 248
  36. 2023Track Anything: Segment Anything Meets Videoscited by 230
  37. 2021Hypercorrelation Squeeze for Few-Shot Segmentationcited by 193
  38. 2023SegGPT: Segmenting Everything In Contextcited by 193

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 39; 2024: 82; 2025: 70; 2026: 59.

250 papers.

Sort:

By area:

  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  2. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Modelcited by 180
  5. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
  6. 2023SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimationcited by 115
  7. 2023SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentationcited by 103
  8. 2023The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shotcited by 83
  9. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  10. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 74
  11. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  12. 2024SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentationcited by 63
38 more
  1. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  2. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  3. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  4. 2023Visual In-Context Promptingcited by 52
  5. 2024Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruningcited by 46
  6. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  7. 2023Annotation-free Audio-Visual Segmentationcited by 44
  8. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  9. 2023MobileSAMv2: Faster Segment Anything to Everythingcited by 41
  10. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  11. 2024Bridge the Points: Graph-based Few-shot Segment Anything Semanticallycited by 35
  12. 2024Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Datasetcited by 33
  13. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  14. 2024DarkSAM: Fooling Segment Anything Model to Segment Nothingcited by 32
  15. 2025Detect Anything 3D in the Wildcited by 32
  16. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  17. 2023Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptationcited by 31
  18. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  19. 2024Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentationcited by 30
  20. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  21. 2024CAT-SAM: Conditional Tuning for Few-Shot Adaptation of Segment Anything Modelcited by 27
  22. 2024APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentationcited by 26
  23. 2023RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentationcited by 24
  24. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24
  25. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 24
  26. 2023Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detectioncited by 22
  27. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  28. 2023A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Applicationcited by 21
  29. 2023SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentationcited by 19
  30. 2023FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detectioncited by 19
  31. 2024A Simple Image Segmentation Framework via In-Context Examplescited by 19
  32. 2025SAMRefiner: Taming Segment Anything Model for Universal Mask Refinementcited by 19
  33. 2023PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentationcited by 18
  34. 2024Where's Waldo: Diffusion Features for Personalized Segmentation and Retrievalcited by 18
  35. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  36. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  37. 2025LENS: Learning to Segment Anything with Unified Reinforced Reasoningcited by 18
  38. 2024PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentationcited by 17

50 most cited of 250 All 250 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision and Language Reference Prompt into SAM for Few-shot Segmentation
  2. 2025X-SAM: From Segment Anything to Any Segmentation
  3. 2025Fast SAM2 with Text-Driven Token Pruning
  4. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
  5. 2025Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation
  6. 2025Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild
  7. 2025The SAM2-to-SAM3 Gap in the Segment Anything Model Family: Why Prompt-Based Expertise Fails in Concept-Driven Image Segmentation
  8. 2025BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts