# IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration

arXiv: https://arxiv.org/abs/2111.09624 · HTML: https://mlsanity.com/paper/2111.09624/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 18 Nov 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features](https://mlsanity.com/paper/2003.03164/) (2020)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration](https://mlsanity.com/paper/2011.12149/) (2020)
- [PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors](https://mlsanity.com/paper/1808.10322/) (2018)
- [Feature-metric Registration: A Fast Semi-supervised Approach for Robust Point Cloud Registration without Correspondences](https://mlsanity.com/paper/2005.01014/) (2020)
- [A comprehensive survey on point cloud registration](https://mlsanity.com/paper/2103.02690/) (2021)
- [End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds](https://mlsanity.com/paper/2003.05855/) (2020)
- [Learning Compact Geometric Features](https://mlsanity.com/paper/1709.05056/) (2017)
- [Visual Explanations From Deep 3D Convolutional Neural Networks for Alzheimer's Disease Classification](https://mlsanity.com/paper/1803.02544/) (2018)
- [A coarse-to-fine algorithm for registration in 3D street-view cross-source point clouds](https://mlsanity.com/paper/1610.07324/) (2016)
- [Fast Registration for cross-source point clouds by using weak regional affinity and pixel-wise refinement](https://mlsanity.com/paper/1903.04630/) (2019)

## Built on by (most cited first, 12 of 12)

- [A Conditional Denoising Diffusion Probabilistic Model for Point Cloud Upsampling](https://mlsanity.com/paper/2312.02719/) (2023)
- [An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Models](https://mlsanity.com/paper/2411.16308/) (2024)
- [PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registration](https://mlsanity.com/paper/2308.04782/) (2023)
- [Cross-source Point Cloud Registration: Challenges, Progress and Prospects](https://mlsanity.com/paper/2305.13570/) (2023)
- [CMHANet: A Cross-Modal Hybrid Attention Network for Point Cloud Registration](https://mlsanity.com/paper/2603.12721/) (2026)
- [GMF: General Multimodal Fusion Framework for Correspondence Outlier Rejection](https://mlsanity.com/paper/2211.00207/) (2022)
- [IGASA: Integrated Geometry-Aware and Skip-Attention Modules for Enhanced Point Cloud Registration](https://mlsanity.com/paper/2603.12719/) (2026)
- [Cross-Modal Information-Guided Network using Contrastive Learning for Point Cloud Registration](https://mlsanity.com/paper/2311.01202/) (2023)
- [Iterative Feedback Network for Unsupervised Point Cloud Registration](https://mlsanity.com/paper/2401.04357/) (2024)
- [Matching Distance and Geometric Distribution Aided Learning Multiview Point Cloud Registration](https://mlsanity.com/paper/2505.03692/) (2025)
- [Cross-Modal Coordination Across a Diverse Set of Input Modalities](https://mlsanity.com/paper/2401.16347/) (2024)
- [T2LDM++: A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation](https://mlsanity.com/paper/2606.30147/) (2026)

## Abstract (arXiv)

The existing state-of-the-art point descriptor relies on structure information only, which omit the texture information. However, texture information is crucial for our humans to distinguish a scene part. Moreover, the current learning-based point descriptors are all black boxes which are unclear how the original points contribute to the final descriptor. In this paper, we propose a new multimodal fusion method to generate a point cloud registration descriptor by considering both structure and texture information. Specifically, a novel attention-fusion module is designed to extract the weighted texture information for the descriptor extraction. In addition, we propose an interpretable module to explain the original points in contributing to the final descriptor. We use the descriptor element as the loss to backpropagate to the target layer and consider the gradient as the significance of this point to the final descriptor. This paper moves one step further to explainable deep learning in the registration task. Comprehensive experiments on 3DMatch, 3DLoMatch and KITTI demonstrate that the multimodal fusion descriptor achieves state-of-the-art accuracy and improve the descriptor's distinctiveness. We also demonstrate that our interpretable module in explaining the registration descriptor extraction.