mlsanityData to 7 Oct 2026

2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

arXiv 2310.12817 ↗First posted 19 Oct 2023cs.CV cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We present a Multimodal Interlaced Transformer (MIT) that jointly considers 2D and 3D data for weakly supervised point cloud segmentation. Research studies have shown that 2D and 3D features are complementary for point cloud segmentation. However, existing methods require extra 2D annotations to achieve 2D-3D information fusion. Considering the high annotation cost of point clouds, effective 2D and 3D feature fusion based on weakly supervised learning is in great demand. To this end, we propose a transformer model with two encoders and one decoder for weakly supervised point cloud segmentation using only scene-level class tags. Specifically, the two encoders compute the self-attended features for 3D point clouds and 2D multi-view images, respectively. The decoder implements interlaced 2D-3D cross-attention and carries out implicit 2D and 3D feature fusion. We alternately switch the roles of queries and key-value pairs in the decoder layers. It turns out that the 2D and 3D features are iteratively enriched by each other. Experiments show that it performs favorably against existing weakly supervised point cloud segmentation methods by a large margin on the S3DIS and ScanNet benchmarks. The project page will be available at https://jimmy15923.github.io/mit_web/.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

49 of 68 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  12. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
37 more
  1. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  2. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  3. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  4. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  5. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  6. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  7. 2020Weakly-Supervised Semantic Segmentation via Sub-category Explorationcited by 145
  8. 2020Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentationcited by 144
  9. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  10. 2020ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votescited by 119
  11. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  12. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  13. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  14. 2021SetVAE: Learning Hierarchical Composition for Generative Modeling of Set-Structured Datacited by 78
  15. 2019Multi-view PointNet for 3D Scene Understandingcited by 76
  16. 2020Multi-Path Region Mining For Weakly Supervised 3D Semantic Segmentation on Point Cloudscited by 71
  17. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  18. 2022Scribble-Supervised LiDAR Semantic Segmentationcited by 69
  19. 20203D Point Cloud Processing and Learning for Autonomous Drivingcited by 68
  20. 2021SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Cloudscited by 64
  21. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  22. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  23. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47
  24. 2020Weakly Supervised Semantic Point Cloud Segmentation:Towards 10X Fewer Labelscited by 45
  25. 2022Large Loss Matters in Weakly Supervised Multi-Label Classificationcited by 38
  26. 20213D Spatial Recognition without Spatially Labeled 3Dcited by 31
  27. 2020Attention-based Multi-modal Fusion Network for Semantic Scene Completioncited by 28
  28. 2022Weakly Supervised Semantic Segmentation for Large-Scale Point Cloudcited by 27
  29. 2021Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistencycited by 19
  30. 2022Bridged Transformer for Vision and Point Cloud 3D Object Detectioncited by 19
  31. 2022Dual Adaptive Transformations for Weakly Supervised Point Cloud Segmentationcited by 15
  32. 2022Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learningcited by 15
  33. 2022SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentationcited by 11
  34. 20203D Guided Weakly Supervised Semantic Segmentationcited by 10
  35. 2022Data Efficient 3D Learner via Knowledge Transferred from 2D Modelcited by 10
  36. 2022GaIA: Graphical Information Gain based Attention Network for Weakly Supervised Point Cloud Semantic Segmentationcited by 9
  37. 2022Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 10; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillationcited by 27
  2. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  3. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  4. 2025BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysiscited by 5
  5. 2025LiDAR Remote Sensing Meets Weak Supervision: Concepts, Methods, and Perspectivescited by 4
  6. 2025Sketchy Bounding-box Supervision for 3D Instance Segmentationcited by 4
  7. 2025HOMER: Homography-Based Efficient Multi-view 3D Object Removalcited by 2
  8. 2024CP-VoteNet: Contrastive Prototypical VoteNet for Few-Shot Point Cloud Object Detectioncited by 1
  9. 2025MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Imagecited by 1
  10. 2025High-quality Pseudo-labeling for Point Cloud Segmentation with Scene-level Annotationcited by 1
  11. 20253D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinementcited by 1
  12. 2023Weakly Supervised Point Cloud Segmentation via Conservative Propagation of Scene-level Labelscited by 0
4 more
  1. 2025Towards Training-Free Open-World Classification with 3D Generative Modelscited by 0
  2. 2025Automating Infrastructure Surveying: A Framework for Geometric Measurements and Compliance Assessment Using Point Cloud Datacited by 0
  3. 2025DBGroup: Dual-Branch Point Grouping for Weakly Supervised 3D Semantic Instance Segmentationcited by 0
  4. 2026G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentationcited by 0