# E-SAM: Training-Free Segment Every Entity Model

arXiv: https://arxiv.org/abs/2503.12094 · HTML: https://mlsanity.com/paper/2503.12094/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 59 references that are in the map.
- First posted on arXiv 15 Mar 2025.

## Builds on (most cited first, 30 of 43)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Segment Anything Meets Point Tracking](https://mlsanity.com/paper/2307.01197/) (2023)
- [Open-World Entity Segmentation](https://mlsanity.com/paper/2107.14228/) (2021)
- [Adaptive NMS: Refining Pedestrian Detection in a Crowd](https://mlsanity.com/paper/1904.03629/) (2019)
- [Mask Transfiner for High-Quality Instance Segmentation](https://mlsanity.com/paper/2111.13673/) (2021)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [Segment Anything Model for Medical Images?](https://mlsanity.com/paper/2304.14660/) (2023)
- [SAM-guided Graph Cut for 3D Instance Segmentation](https://mlsanity.com/paper/2312.08372/) (2023)
- [TinySAM: Pushing the Envelope for Efficient Segment Anything Model](https://mlsanity.com/paper/2312.13789/) (2023)
- [UniGS: Unified Representation for Image Generation and Segmentation](https://mlsanity.com/paper/2312.01985/) (2023)
- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [Segment Anything without Supervision](https://mlsanity.com/paper/2406.20081/) (2024)
- [MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing](https://mlsanity.com/paper/2402.14835/) (2024)

## Built on by (most cited first, 3 of 3)

- [Leader360V: The Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment](https://mlsanity.com/paper/2506.14271/) (2025)
- [Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagery](https://mlsanity.com/paper/2601.21159/) (2026)
- [A Geometric-Transformation Feature-Adaptive Manifold Restoration Method for Open-Vocabulary Semantic Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2610.04300/) (2026)

## Abstract (arXiv)

Entity Segmentation (ES) aims at identifying and segmenting distinct entities within an image without the need for predefined class labels. This characteristic makes ES well-suited to open-world applications with adaptation to diverse and dynamically changing environments, where new and previously unseen entities may appear frequently. Existing ES methods either require large annotated datasets or high training costs, limiting their scalability and adaptability. Recently, the Segment Anything Model (SAM), especially in its Automatic Mask Generation (AMG) mode, has shown potential for holistic image segmentation. However, it struggles with over-segmentation and under-segmentation, making it less effective for ES. In this paper, we introduce E-SAM, a novel training-free framework that exhibits exceptional ES capability. Specifically, we first propose Multi-level Mask Generation (MMG) that hierarchically processes SAM's AMG outputs to generate reliable object-level masks while preserving fine details at other levels. Entity-level Mask Refinement (EMR) then refines these object-level masks into accurate entity-level masks. That is, it separates overlapping masks to address the redundancy issues inherent in SAM's outputs and merges similar masks by evaluating entity-level consistency. Lastly, Under-Segmentation Refinement (USR) addresses under-segmentation by generating additional high-confidence masks fused with EMR outputs to produce the final ES map. These three modules are seamlessly optimized to achieve the best ES without additional training overhead. Extensive experiments demonstrate that E-SAM achieves state-of-the-art performance compared to prior ES methods, demonstrating a significant improvement by +30.1 on benchmark metrics.