mlsanityData to 14 Oct 2026

3D Instance Segmentation of MVS Buildings

arXiv 2112.09902 ↗First posted 18 Dec 2021cs.CV cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

We present a novel 3D instance segmentation framework for Multi-View Stereo (MVS) buildings in urban scenes. Unlike existing works focusing on semantic segmentation of urban scenes, the emphasis of this work lies in detecting and segmenting 3D building instances even if they are attached and embedded in a large and imprecise 3D surface model. Multi-view RGB images are first enhanced to RGBH images by adding a heightmap and are segmented to obtain all roof instances using a fine-tuned 2D instance segmentation neural network. Instance masks from different multi-view images are then clustered into global masks. Our mask clustering accounts for spatial occlusion and overlapping, which can eliminate segmentation ambiguities among multi-view images. Based on these global masks, 3D roof instances are segmented out by mask back-projections and extended to the entire building instances through a Markov random field optimization. A new dataset that contains instance-level annotation for both 3D urban scenes (roofs and buildings) and drone images (roofs) is provided. To the best of our knowledge, it is the first outdoor dataset dedicated to 3D instance segmentation with much more annotations of attached 3D buildings than existing datasets. Quantitative evaluations and ablation studies have shown the effectiveness of all major steps and the advantages of our multi-view framework over the orthophoto-based method.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 51 of its 78 references that are in the map. First posted on arXiv 18 Dec 2021.

Builds on

51 of 78 references.

References · 51

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2017Mask R-CNNcited by 7,277
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  9. 2015Fast R-CNNcited by 2,933
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  11. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  12. 2018Path Aggregation Network for Instance Segmentationcited by 798
38 more
  1. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  2. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  3. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 720
  4. 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 623
  5. 2019Hybrid Task Cascade for Instance Segmentationcited by 481
  6. 2019YOLACT: Real-time Instance Segmentationcited by 443
  7. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 342
  8. 2019Mask Scoring R-CNNcited by 266
  9. 2020Detection and Tracking Meet Drones Challengecited by 263
  10. 2016Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesiscited by 219
  11. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 217
  12. 2018SpaceNet: A Remote Sensing Dataset and Challenge Seriescited by 179
  13. 2017SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentationcited by 169
  14. 2021SpectralFormer: Rethinking Hyperspectral Image Classification with Transformerscited by 169
  15. 2018xView: Objects in Context in Overhead Imagerycited by 146
  16. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 146
  17. 2020BlendMask: Top-Down Meets Bottom-Up for Instance Segmentationcited by 145
  18. 2020More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classificationcited by 126
  19. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 123
  20. 2017Semantic Instance Segmentation with a Discriminative Loss Functioncited by 118
  21. 2020Graph Convolutional Networks for Hyperspectral Image Classificationcited by 113
  22. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  23. 2019Associatively Segmenting Instances and Semantics in Point Cloudscited by 86
  24. 2016Deep Watershed Transform for Instance Segmentationcited by 78
  25. 2018An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixingcited by 66
  26. 2019JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fieldscited by 62
  27. 2017Recurrent Pixel Embedding for Instance Groupingcited by 58
  28. 2018Topological Map Extraction from Overhead Imagescited by 58
  29. 2017O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysiscited by 47
  30. 2017Semantic Instance Segmentation via Deep Metric Learningcited by 37
  31. 2021The Hessigheim 3D (H3D) Benchmark on Semantic Segmentation of High-Resolution 3D Point Clouds and Textured Meshes from UAV LiDAR and Multi-View-Stereocited by 37
  32. 2019DARNet: Deep Active Ray Network for Building Segmentationcited by 30
  33. 2019JSNet: Joint Instance and Semantic Segmentation of 3D Point Cloudscited by 27
  34. 2019MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentationcited by 25
  35. 2019Conv-MPN: Convolutional Message Passing Neural Network for Structured Outdoor Architecture Reconstructioncited by 25
  36. 2021SUM: A Benchmark Dataset of Semantic Urban Meshescited by 23
  37. 2018Semi-convolutional Operators for Instance Segmentationcited by 22
  38. 2020Semantic Segmentation on Swiss3DCities: A Benchmark Study on Aerial Photogrammetric 3D Pointcloud Datasetcited by 22

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 2; 2026: 2.

12 papers.

  1. 2023Segment Anythingcited by 8,234
  2. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  3. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  4. 2023UrbanBIS: a Large-scale Benchmark for Fine-grained Urban Building Instance Segmentationcited by 24
  5. 2024ZIM: Zero-Shot Image Matting for Anythingcited by 16
  6. 2025SAM2 for Image and Video Segmentation: A Comprehensive Surveycited by 13
  7. 2024CRASAR-U-DROIDs: A Large Scale Benchmark Dataset for Building Alignment and Damage Assessment in Georectified sUAS Imagerycited by 9
  8. 2024Aerial Lifting: Neural Urban Semantic and Building Instance Lifting from Aerial Imagerycited by 2
  9. 2023EipFormer: Emphasizing Instance Positions in 3D Instance Segmentationcited by 1
  10. 2024SCLNet: A Scale-Robust Complementary Learning Network for Object Detection in UAV Imagescited by 1
  11. 2026OutLangSplat: 3D Language Gaussian Splatting for UAV Outdoor Scenescited by 1
  12. 2026Segment Anything with Robust Uncertainty-Accuracy Correlationcited by 0