mlsanityData to 14 Oct 2026

Joint Depth Prediction and Semantic Segmentation with Multi-View SAM

arXiv 2311.00134 ↗First posted 31 Oct 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Multi-task approaches to joint depth and segmentation prediction are well-studied for monocular images. Yet, predictions from a single-view are inherently limited, while multiple views are available in many robotics applications. On the other end of the spectrum, video-based and full 3D methods require numerous frames to perform reconstruction and segmentation. With this work we propose a Multi-View Stereo (MVS) technique for depth prediction that benefits from rich semantic features of the Segment Anything Model (SAM). This enhanced depth prediction, in turn, serves as a prompt to our Transformer-based semantic segmentation decoder. We report the mutual benefit that both tasks enjoy in our quantitative and qualitative studies on the ScanNet dataset. Our approach consistently outperforms single-task MVS and segmentation models, along with multi-task monocular methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

48 of 64 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2023Segment Anythingcited by 8,234
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2016Pyramid Scene Parsing Networkcited by 2,479
  10. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  11. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  12. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
36 more
  1. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  2. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  3. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
  4. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  5. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  6. 2019Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matchingcited by 417
  7. 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
  8. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  9. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  10. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  11. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  12. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  13. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  14. 2020Semantically-Guided Representation Learning for Self-Supervised Monocular Depthcited by 134
  15. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  16. 2022DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimationcited by 118
  17. 2021TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformerscited by 117
  18. 2019Multi-Scale Geometric Consistency Guided Multi-View Stereocited by 105
  19. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  20. 2022Attention Attention Everywhere: Monocular Depth Prediction with Skip Attentioncited by 93
  21. 2018SegStereo: Exploiting Semantic Information for Disparity Estimationcited by 88
  22. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  23. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  24. 2023iDisc: Internal Discretization for Monocular Depth Estimationcited by 73
  25. 2020The Edge of Depth: Explicit Constraints between Segmentation and Depthcited by 72
  26. 2020DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusioncited by 68
  27. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 66
  28. 2023VA-DepthNet: A Variational Approach to Single Image Depth Predictioncited by 54
  29. 2018Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotationscited by 51
  30. 2019Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semanticscited by 49
  31. 2021MVS2D: Efficient Multi-view Stereo via Attention-Driven 2D Convolutionscited by 43
  32. 2021SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular imagescited by 33
  33. 2021Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentationcited by 31
  34. 2023MVPSNet: Fast Generalizable Multi-view Photometric Stereocited by 18
  35. 2023TSAR-MVS: Textureless-aware Segmentation and Correlative Refinement Guided Multi-View Stereocited by 9
  36. 2023Efficient Multi-Task Scene Analysis with RGB-D Transformerscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 3.

8 papers.

  1. 2024Learning-based Multi-View Stereo: A Surveycited by 18
  2. 2024MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereocited by 17
  3. 2024SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentationcited by 6
  4. 2024SMART: Scene-motion-aware human action recognition framework for mental disorder groupcited by 3
  5. 2024SAMReg: SAM-enabled Image Registration with ROI-based Correspondencecited by 2
  6. 2025EndoUFM: Utilizing Foundation Models for Monocular depth estimation of endoscopic imagescited by 1
  7. 2025Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsingcited by 0
  8. 2025Bridging Geometric and Semantic Foundation Models for Generalized Monocular Depth Estimationcited by 0