# RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for Real-Time Point Cloud Shape Completion

arXiv: https://arxiv.org/abs/1904.12304 · HTML: https://mlsanity.com/paper/1904.12304/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2019.

## Built on by (most cited first, 30 of 58)

- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers](https://mlsanity.com/paper/2301.04545/) (2023)
- [PMP-Net++: Point Cloud Completion by Transformer-Enhanced Multi-step Point Moving Paths](https://mlsanity.com/paper/2202.09507/) (2022)
- [Cycle4Completion: Unpaired Point Cloud Completion using Cycle Transformation with Missing Region Coding](https://mlsanity.com/paper/2103.07838/) (2021)
- [Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis](https://mlsanity.com/paper/2203.03311/) (2022)
- [PointAttN: You Only Need Attention for Point Cloud Completion](https://mlsanity.com/paper/2203.08485/) (2022)
- [Voxel-based Network for Shape Completion by Leveraging Edge Generation](https://mlsanity.com/paper/2108.09936/) (2021)
- [Orienting Point Clouds with Dipole Propagation](https://mlsanity.com/paper/2105.01604/) (2021)
- [ASFM-Net: Asymmetrical Siamese Feature Matching Network for Point Completion](https://mlsanity.com/paper/2104.09587/) (2021)
- [CSDN: Cross-modal Shape-transfer Dual-refinement Network for Point Cloud Completion](https://mlsanity.com/paper/2208.00751/) (2022)
- [Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild](https://mlsanity.com/paper/2101.06860/) (2021)
- [Denoise and Contrast for Category Agnostic Shape Completion](https://mlsanity.com/paper/2103.16671/) (2021)
- [AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation](https://mlsanity.com/paper/2409.00342/) (2024)
- [Progressive Seed Generation Auto-encoder for Unsupervised Point Cloud Learning](https://mlsanity.com/paper/2112.05213/) (2021)
- [Automated Urban Planning for Reimagining City Configuration via Adversarial Learning: Quantification, Generation, and Evaluation](https://mlsanity.com/paper/2112.14699/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Unsupervised Point Cloud Pre-Training via Contrasting and Clustering](https://mlsanity.com/paper/2202.02543/) (2022)
- [GeoPointGAN: Synthetic Spatial Data with Local Label Differential Privacy](https://mlsanity.com/paper/2205.08886/) (2022)
- [Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion](https://mlsanity.com/paper/2407.02887/) (2024)
- [Attention-based Transformation from Latent Features to Point Clouds](https://mlsanity.com/paper/2112.05324/) (2021)
- [Game of GANs: Game-Theoretical Models for Generative Adversarial Networks](https://mlsanity.com/paper/2106.06976/) (2021)
- [Graph-Guided Deformation for Point Cloud Completion](https://mlsanity.com/paper/2112.01840/) (2021)
- [Data Augmentation-free Unsupervised Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2210.02798/) (2022)
- [DeepMend: Learning Occupancy Functions to Represent Shape for Repair](https://mlsanity.com/paper/2210.05728/) (2022)
- [Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image](https://mlsanity.com/paper/2210.05891/) (2022)
- [Free-form 3D Scene Inpainting with Dual-stream GAN](https://mlsanity.com/paper/2212.08464/) (2022)
- [Reconstruction-Aware Prior Distillation for Semi-supervised Point Cloud Completion](https://mlsanity.com/paper/2204.09186/) (2022)
- [DeepJoin: Learning a Joint Occupancy, Signed Distance, and Normal Field Function for Shape Repair](https://mlsanity.com/paper/2211.12400/) (2022)
- [Anatomy Completor: A Multi-class Completion Framework for 3D Anatomy Reconstruction](https://mlsanity.com/paper/2309.04956/) (2023)

## Abstract (arXiv)

We present RL-GAN-Net, where a reinforcement learning (RL) agent provides fast and robust control of a generative adversarial network (GAN). Our framework is applied to point cloud shape completion that converts noisy, partial point cloud data into a high-fidelity completed shape by controlling the GAN. While a GAN is unstable and hard to train, we circumvent the problem by (1) training the GAN on the latent space representation whose dimension is reduced compared to the raw point cloud input and (2) using an RL agent to find the correct input to the GAN to generate the latent space representation of the shape that best fits the current input of incomplete point cloud. The suggested pipeline robustly completes point cloud with large missing regions. To the best of our knowledge, this is the first attempt to train an RL agent to control the GAN, which effectively learns the highly nonlinear mapping from the input noise of the GAN to the latent space of point cloud. The RL agent replaces the need for complex optimization and consequently makes our technique real time. Additionally, we demonstrate that our pipelines can be used to enhance the classification accuracy of point cloud with missing data.