# Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation

arXiv: https://arxiv.org/abs/2003.06233 · HTML: https://mlsanity.com/paper/2003.06233/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 30 of 40)

- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation](https://mlsanity.com/paper/2410.08189/) (2024)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2311.15383/) (2023)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [Pri3D: Can 3D Priors Help 2D Representation Learning?](https://mlsanity.com/paper/2104.11225/) (2021)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception](https://mlsanity.com/paper/2109.05441/) (2021)
- [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [EmbodiedSAM: Online Segment Any 3D Thing in Real Time](https://mlsanity.com/paper/2408.11811/) (2024)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation](https://mlsanity.com/paper/2108.07511/) (2021)
- [MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction](https://mlsanity.com/paper/2308.08741/) (2023)
- [Boundary-Aware Geometric Encoding for Semantic Segmentation of Point Clouds](https://mlsanity.com/paper/2101.02381/) (2021)
- [Clustering based Point Cloud Representation Learning for 3D Analysis](https://mlsanity.com/paper/2307.14605/) (2023)
- [Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning](https://mlsanity.com/paper/2105.10203/) (2021)
- [GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion](https://mlsanity.com/paper/2309.15459/) (2023)
- [Point Cloud Learning with Transformer](https://mlsanity.com/paper/2104.13636/) (2021)
- [TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation](https://mlsanity.com/paper/2411.16425/) (2024)
- [Memory-based Adapters for Online 3D Scene Perception](https://mlsanity.com/paper/2403.06974/) (2024)
- [A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenes](https://mlsanity.com/paper/2108.05246/) (2021)
- [LLM-enhanced Scene Graph Learning for Household Rearrangement](https://mlsanity.com/paper/2408.12093/) (2024)
- [OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging](https://mlsanity.com/paper/2503.01309/) (2025)
- [SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentation](https://mlsanity.com/paper/2205.13490/) (2022)
- [Semi-supervised 3D shape segmentation with multilevel consistency and part substitution](https://mlsanity.com/paper/2204.08824/) (2022)
- [Robust Change Detection Based on Neural Descriptor Fields](https://mlsanity.com/paper/2208.01014/) (2022)
- [ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction](https://mlsanity.com/paper/2311.18068/) (2023)
- [Active Self-Training for Weakly Supervised 3D Scene Semantic Segmentation](https://mlsanity.com/paper/2209.07069/) (2022)
- [Dense Supervision Propagation for Weakly Supervised Semantic Segmentation on 3D Point Clouds](https://mlsanity.com/paper/2107.11267/) (2021)
- [PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum](https://mlsanity.com/paper/2506.23607/) (2025)

## Abstract (arXiv)

Online semantic 3D segmentation in company with real-time RGB-D reconstruction poses special challenges such as how to perform 3D convolution directly over the progressively fused 3D geometric data, and how to smartly fuse information from frame to frame. We propose a novel fusion-aware 3D point convolution which operates directly on the geometric surface being reconstructed and exploits effectively the inter-frame correlation for high quality 3D feature learning. This is enabled by a dedicated dynamic data structure which organizes the online acquired point cloud with global-local trees. Globally, we compile the online reconstructed 3D points into an incrementally growing coordinate interval tree, enabling fast point insertion and neighborhood query. Locally, we maintain the neighborhood information for each point using an octree whose construction benefits from the fast query of the global tree.Both levels of trees update dynamically and help the 3D convolution effectively exploits the temporal coherence for effective information fusion across RGB-D frames.