mlsanityData to 7 Oct 2026

Impact of Pseudo Depth on Open World Object Segmentation with Minimal User Guidance

arXiv 2304.05716 ↗First posted 12 Apr 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Pseudo depth maps are depth map predicitions which are used as ground truth during training. In this paper we leverage pseudo depth maps in order to segment objects of classes that have never been seen during training. This renders our object segmentation task an open world task. The pseudo depth maps are generated using pretrained networks, which have either been trained with the full intention to generalize to downstream tasks (LeRes and MiDaS), or which have been trained in an unsupervised fashion on video sequences (MonodepthV2). In order to tell our network which object to segment, we provide the network with a single click on the object's surface on the pseudo depth map of the image as input. We test our approach on two different scenarios: One without the RGB image and one where the RGB image is part of the input. Our results demonstrate a considerably better generalization performance from seen to unseen object types when depth is used. On the Semantic Boundaries Dataset we achieve an improvement from 61.57 to 69.79 IoU score on unseen classes, when only using half of the training classes during training and performing the segmentation on depth maps only.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

42 of 57 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  5. 2021Vision Transformers for Dense Predictioncited by 1,789
  6. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  7. 2015Spatial Transformer Networkscited by 1,423
  8. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  9. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  10. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  11. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
  12. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
30 more
  1. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  2. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  3. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  4. 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
  5. 2018The ApolloScape Open Dataset for Autonomous Driving and its Applicationcited by 188
  6. 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
  7. 2021Reviving Iterative Training with Mask Guidance for Interactive Segmentationcited by 162
  8. 2018Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videoscited by 155
  9. 2020Feature-metric Loss for Self-supervised Learning of Depth and Egomotioncited by 140
  10. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  11. 20193D Ken Burns Effect from a Single Imagecited by 128
  12. 2017Deep Extreme Cut: From Extreme Points to Object Segmentationcited by 127
  13. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  14. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 120
  15. 2020f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentationcited by 117
  16. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  17. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  18. 2022FocalClick: Towards Practical Interactive Image Segmentationcited by 101
  19. 2021Pointly-Supervised Instance Segmentationcited by 83
  20. 2019Web Stereo Video Supervision for Depth Prediction from Dynamic Scenescited by 73
  21. 2021EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flowcited by 40
  22. 2021Learning to Detect Every Thing in an Open Worldcited by 37
  23. 2020Holopix50k: A Large-Scale In-the-wild Stereo Image Datasetcited by 36
  24. 2020Bilateral Attention Network for RGB-D Salient Object Detectioncited by 35
  25. 2019Unsupervised High-Resolution Depth Learning From Videos With Dual Networkscited by 26
  26. 2021Unknown Object Segmentation from Stereo Imagescited by 16
  27. 2019IRS: A Large Naturalistic Indoor Robotics Stereo Dataset to Train Deep Models for Disparity and Surface Normal Estimationcited by 12
  28. 2021UCP-Net: Unstructured Contour Points for Instance Segmentationcited by 7
  29. 2022Attention-based Dual Supervised Decoder for RGBD Semantic Segmentationcited by 6
  30. 2018Object segmentation in depth maps with one user click and a synthetically trained fully convolutional networkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025High-Precision Dichotomous Image Segmentation via Depth Integrity-Prior and Fine-Grained Patch Strategycited by 1
  2. 2025MMMS: Multi-Modal Multi-Surface Interactive Segmentationcited by 0
  3. 2026studentSplat: Your Student Model Learns Single-view 3D Gaussian Splattingcited by 0
  4. 2026Monocular Depth Estimation from a Single Image: Progress and Opportunitiescited by 0