# AoP-SAM: Automation of Prompts for Efficient Segmentation

arXiv: https://arxiv.org/abs/2505.11980 · HTML: https://mlsanity.com/paper/2505.11980/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 17 May 2025.

## Builds on (most cited first, 12 of 12)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering](https://mlsanity.com/paper/2306.06211/) (2023)
- [MobileSAMv2: Faster Segment Anything to Everything](https://mlsanity.com/paper/2312.09579/) (2023)
- [Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects](https://mlsanity.com/paper/2312.07374/) (2023)
- [End-to-end training of object class detectors for mean average precision](https://mlsanity.com/paper/1607.03476/) (2016)
- [Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detected](https://mlsanity.com/paper/2305.00278/) (2023)

## Built on by (most cited first, 7 of 7)

- [Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark](https://mlsanity.com/paper/2508.04260/) (2025)
- [DyABD: The Abdominal Muscle Segmentation in Dynamic MRI Benchmark](https://mlsanity.com/paper/2604.23187/) (2026)
- [Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentation](https://mlsanity.com/paper/2511.19062/) (2025)
- [Disentangling Prompt Dependence to Evaluate Segmentation Reliability in Gynecological MRI](https://mlsanity.com/paper/2603.13369/) (2026)
- [IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment](https://mlsanity.com/paper/2603.27250/) (2026)
- [SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM](https://mlsanity.com/paper/2609.18037/) (2026)
- [SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation](https://mlsanity.com/paper/2609.24226/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM) is a powerful foundation model for image segmentation, showing robust zero-shot generalization through prompt engineering. However, relying on manual prompts is impractical for real-world applications, particularly in scenarios where rapid prompt provision and resource efficiency are crucial. In this paper, we propose the Automation of Prompts for SAM (AoP-SAM), a novel approach that learns to generate essential prompts in optimal locations automatically. AoP-SAM enhances SAM's efficiency and usability by eliminating manual input, making it better suited for real-world tasks. Our approach employs a lightweight yet efficient Prompt Predictor model that detects key entities across images and identifies the optimal regions for placing prompt candidates. This method leverages SAM's image embeddings, preserving its zero-shot generalization capabilities without requiring fine-tuning. Additionally, we introduce a test-time instance-level Adaptive Sampling and Filtering mechanism that generates prompts in a coarse-to-fine manner. This notably enhances both prompt and mask generation efficiency by reducing computational overhead and minimizing redundant mask refinements. Evaluations of three datasets demonstrate that AoP-SAM substantially improves both prompt generation efficiency and mask generation accuracy, making SAM more effective for automated segmentation tasks.