# Cross-modal Learning for Image-Guided Point Cloud Shape Completion

arXiv: https://arxiv.org/abs/2209.09552 · HTML: https://mlsanity.com/paper/2209.09552/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [Self-Attention Graph Pooling](https://mlsanity.com/paper/1904.08082/) (2019)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis](https://mlsanity.com/paper/1612.00101/) (2016)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [GRNet: Gridding Residual Network for Dense Point Cloud Completion](https://mlsanity.com/paper/2006.03761/) (2020)
- [SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer](https://mlsanity.com/paper/2108.04444/) (2021)
- [PF-Net: Point Fractal Network for 3D Point Cloud Completion](https://mlsanity.com/paper/2003.00410/) (2020)
- [Morphing and Sampling Network for Dense Point Cloud Completion](https://mlsanity.com/paper/1912.00280/) (2019)
- [Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review](https://mlsanity.com/paper/2005.09830/) (2020)
- [Variational Relational Point Completion Network](https://mlsanity.com/paper/2104.10154/) (2021)
- [Unsupervised Learning of Shape and Pose with Differentiable Point Clouds](https://mlsanity.com/paper/1810.09381/) (2018)
- [High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference](https://mlsanity.com/paper/1709.07599/) (2017)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [View-Guided Point Cloud Completion](https://mlsanity.com/paper/2104.05666/) (2021)
- [Style-based Point Generator with Adversarial Rendering for Point Cloud Completion](https://mlsanity.com/paper/2103.02535/) (2021)
- [Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis](https://mlsanity.com/paper/2203.03311/) (2022)
- [Cascaded Refinement Network for Point Cloud Completion with Self-supervision](https://mlsanity.com/paper/2010.08719/) (2020)
- [Denoise and Contrast for Category Agnostic Shape Completion](https://mlsanity.com/paper/2103.16671/) (2021)
- [Self-Supervised Point Cloud Completion via Inpainting](https://mlsanity.com/paper/2111.10701/) (2021)
- [Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild](https://mlsanity.com/paper/2007.08939/) (2020)
- [Attention-based Transformation from Latent Features to Point Clouds](https://mlsanity.com/paper/2112.05324/) (2021)

## Built on by (most cited first, 30 of 36)

- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [Point-Cloud Completion with Pretrained Text-to-image Diffusion Models](https://mlsanity.com/paper/2306.10533/) (2023)
- [SVDFormer: Complementing Point Cloud via Self-view Augmentation and Self-structure Dual-generator](https://mlsanity.com/paper/2307.08492/) (2023)
- [GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding](https://mlsanity.com/paper/2411.19626/) (2024)
- [Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding](https://mlsanity.com/paper/2408.13024/) (2024)
- [SuperPC: A Single Diffusion Model for Point Cloud Completion, Upsampling, Denoising, and Colorization](https://mlsanity.com/paper/2503.14558/) (2025)
- [InteractVLM: 3D Interaction Reasoning from 2D Foundational Models](https://mlsanity.com/paper/2504.05303/) (2025)
- [GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency](https://mlsanity.com/paper/2412.09511/) (2024)
- [PCDreamer: Point Cloud Completion Through Multi-view Diffusion Priors](https://mlsanity.com/paper/2411.19036/) (2024)
- [Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion](https://mlsanity.com/paper/2407.02887/) (2024)
- [PointSea: Point Cloud Completion via Self-structure Augmentation](https://mlsanity.com/paper/2502.17053/) (2025)
- [Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion](https://mlsanity.com/paper/2212.00564/) (2022)
- [Unsupervised 3D Point Cloud Completion via Multi-view Adversarial Learning](https://mlsanity.com/paper/2407.09786/) (2024)
- [Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning](https://mlsanity.com/paper/2508.01651/) (2025)
- [Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model](https://mlsanity.com/paper/2308.08754/) (2023)
- [DMF-Net: Image-Guided Point Cloud Completion with Dual-Channel Modality Fusion and Shape-Aware Upsampling Transformer](https://mlsanity.com/paper/2406.17319/) (2024)
- [DiffPCN: Latent Diffusion Model Based on Multi-view Depth Images for Point Cloud Completion](https://mlsanity.com/paper/2509.23723/) (2025)
- [Deep Learning for 3D Point Cloud Enhancement: A Survey](https://mlsanity.com/paper/2411.00857/) (2024)
- [MambaTron: Efficient Cross-Modal Point Cloud Enhancement using Aggregate Selective State Space Modeling](https://mlsanity.com/paper/2501.16384/) (2025)
- [Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge](https://mlsanity.com/paper/2510.08316/) (2025)
- [MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding](https://mlsanity.com/paper/2601.03660/) (2026)
- [DuInNet: Dual-Modality Feature Interaction for Point Cloud Completion](https://mlsanity.com/paper/2407.07374/) (2024)
- [Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion](https://mlsanity.com/paper/2507.14485/) (2025)
- [Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning](https://mlsanity.com/paper/2508.01184/) (2025)
- [HeartFormer: Semantic-Aware Dual-Structure Transformers for 3D Four-Chamber Cardiac Point Cloud Reconstruction](https://mlsanity.com/paper/2512.00264/) (2025)
- [VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos](https://mlsanity.com/paper/2602.20608/) (2026)
- [HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding](https://mlsanity.com/paper/2603.02329/) (2026)
- [SplAttN: Bridging 2D and 3D with Gaussian Soft Splatting and Attention for Point Cloud Completion](https://mlsanity.com/paper/2605.01466/) (2026)
- [I2PRef: Image-Driven Point Completion with Iterative Refinement](https://mlsanity.com/paper/2605.26914/) (2026)
- [MASSTAR: A Multi-Modal and Large-Scale Scene Dataset with a Versatile Toolchain for Surface Prediction and Completion](https://mlsanity.com/paper/2403.11681/) (2024)

## Abstract (arXiv)

In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significant improvements over state-of-the-art supervised methods for both unimodal and multimodal completion. We also show the effectiveness of the weakly-supervised approach which outperforms a number of supervised methods and is competitive with the latest supervised models only exploiting point cloud information.