mlsanityData to 7 Oct 2026

Lightweight integration of 3D features to improve 2D image segmentation

arXiv 2212.08334 ↗First posted 16 Dec 2022cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Scene understanding has made tremendous progress over the past few years, as data acquisition systems are now providing an increasing amount of data of various modalities (point cloud, depth, RGB...). However, this improvement comes at a large cost on computation resources and data annotation requirements. To analyze geometric information and images jointly, many approaches rely on both a 2D loss and 3D loss, requiring not only 2D per pixel-labels but also 3D per-point labels. However, obtaining a 3D groundtruth is challenging, time-consuming and error-prone. In this paper, we show that image segmentation can benefit from 3D geometric information without requiring a 3D groundtruth, by training the geometric feature extraction and the 2D segmentation network jointly, in an end-to-end fashion, using only the 2D segmentation loss. Our method starts by extracting a map of 3D features directly from a provided point cloud by using a lightweight 3D neural network. The 3D feature map, merged with the RGB image, is then used as an input to a classical image segmentation network. Our method can be applied to many 2D segmentation networks, improving significantly their performance with only a marginal network weight increase and light input dataset requirements, since no 3D groundtruth is required.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 36 of its 41 references that are in the map. First posted on arXiv 16 Dec 2022.

Builds on

36 of 41 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
24 more
  1. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  2. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  3. 2017Deformable Convolutional Networkscited by 1,481
  4. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  6. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  8. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  9. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  10. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  11. 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
  12. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  13. 2018SPLATNet: Sparse Lattice Networks for Point Cloud Processingcited by 158
  14. 2017Pointwise Convolutional Neural Networkscited by 126
  15. 2014Learning Rich Features from RGB-D Images for Object Detection and Segmentationcited by 124
  16. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  17. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  18. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  19. 2019Multi-view PointNet for 3D Scene Understandingcited by 76
  20. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 75
  21. 2018Depth-aware CNN for RGB-D Segmentationcited by 73
  22. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  23. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 47
  24. 2019RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentationcited by 12

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds from RGB Images for 2D Classificationcited by 2