mlsanityData to 14 Oct 2026

SOS: Segment Object System for Open-World Instance Segmentation With Object Priors

arXiv 2409.14627 ↗First posted 22 Sep 2024cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

We propose an approach for Open-World Instance Segmentation (OWIS), a task that aims to segment arbitrary unknown objects in images by generalizing from a limited set of annotated object classes during training. Our Segment Object System (SOS) explicitly addresses the generalization ability and the low precision of state-of-the-art systems, which often generate background detections. To this end, we generate high-quality pseudo annotations based on the foundation model SAM. We thoroughly study various object priors to generate prompts for SAM, explicitly focusing the foundation model on objects. The strongest object priors were obtained by self-attention maps from self-supervised Vision Transformers, which we utilize for prompting SAM. Finally, the post-processed segments from SAM are used as pseudo annotations to train a standard instance segmentation system. Our approach shows strong generalization capabilities on COCO, LVIS, and ADE20k datasets and improves on the precision by up to 81.6% compared to the state-of-the-art. Source code is available at: https://github.com/chwilms/SOS

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 22 Sep 2024.

Builds on

45 of 62 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2023Segment Anythingcited by 8,234
  10. 2017Mask R-CNNcited by 7,277
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
33 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2015Fast R-CNNcited by 2,933
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  7. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  8. 2019Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relationscited by 247
  9. 2019EGNet:Edge Guidance Network for Salient Object Detectioncited by 237
  10. 2016Deeply supervised salient object detection with short connectionscited by 196
  11. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  12. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  13. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  14. 2020Unseen Object Instance Segmentation for Robotic Environmentscited by 85
  15. 2015Learning to Segment Object Candidatescited by 79
  16. 2016Learning to Refine Object Segmentscited by 67
  17. 2021DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modelingcited by 56
  18. 2023Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentationcited by 53
  19. 2022Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinitycited by 41
  20. 2021Learning to Detect Every Thing in an Open Worldcited by 37
  21. 2023Training-free Object Counting with Promptscited by 34
  22. 2023Foundation Model Assisted Weakly Supervised Semantic Segmentationcited by 33
  23. 2021Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinementcited by 26
  24. 2023SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learningcited by 17
  25. 2023Semantic-aware SAM for Point-Prompted Instance Segmentationcited by 16
  26. 2016FastMask: Segment Multi-scale Object Candidates in One Shotcited by 10
  27. 2018AttentionMask: Attentive, Efficient Object Proposal Generation Focusing on Small Objectscited by 9
  28. 2023Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervisioncited by 8
  29. 2023Zero-Shot Anomaly Detection with Pre-trained Segmentation Modelscited by 8
  30. 2023Exploring Transformers for Open-world Instance Segmentationcited by 5
  31. 2022Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularizationcited by 4
  32. 2023OpenInst: A Simple Query-Based Method for Open-World Instance Segmentationcited by 4
  33. 2017Saliency-guided Adaptive Seeding for Supervoxel Segmentationcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Towards 3D Objectness Learning in an Open Worldcited by 2
  2. 2025v-CLR: View-Consistent Learning for Open-World Instance Segmentationcited by 1
  3. 2025SegSLR: Promptable Video Segmentation for Isolated Sign Language Recognitioncited by 0