# Dynamic Object Removal and Spatio-Temporal RGB-D Inpainting via Geometry-Aware Adversarial Learning

arXiv: https://arxiv.org/abs/2008.05058 · HTML: https://mlsanity.com/paper/2008.05058/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2020.

## Built on by (most cited first, 19 of 19)

- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior](https://mlsanity.com/paper/2404.11613/) (2024)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [Unsupervised Domain Adaptation for LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2109.15286/) (2021)
- [CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation](https://mlsanity.com/paper/2303.10147/) (2023)
- [CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration](https://mlsanity.com/paper/2402.00129/) (2024)
- [A Point-Based Approach to Efficient LiDAR Multi-Task Perception](https://mlsanity.com/paper/2404.12798/) (2024)
- [Semantic Visual Simultaneous Localization and Mapping: A Survey](https://mlsanity.com/paper/2209.06428/) (2022)
- [On Hyperbolic Embeddings in 2D Object Detection](https://mlsanity.com/paper/2203.08049/) (2022)
- [Visual-Inertial SLAM for Unstructured Outdoor Environments: Benchmarking the Benefits and Computational Costs of Loop Closing](https://mlsanity.com/paper/2408.01716/) (2024)
- [ROVER: A Multi-Season Dataset for Visual SLAM](https://mlsanity.com/paper/2412.02506/) (2024)
- [DSLR: Dynamic to Static LiDAR Scan Reconstruction Using Adversarially Trained Autoencoder](https://mlsanity.com/paper/2105.12774/) (2021)
- [One-Shot Video Inpainting](https://mlsanity.com/paper/2302.14362/) (2023)
- [DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality](https://mlsanity.com/paper/2312.00532/) (2023)
- [Multi-Perspective Anomaly Detection](https://mlsanity.com/paper/2105.09903/) (2021)
- [7th AI Driving Olympics: 1st Place Report for Panoptic Tracking](https://mlsanity.com/paper/2112.05210/) (2021)
- [AnchorD: Metric Grounding of Monocular Depth Using Factor Graphs](https://mlsanity.com/paper/2605.02667/) (2026)
- [Mask-Guided Image Person Removal with Data Synthesis](https://mlsanity.com/paper/2209.14890/) (2022)
- [A Real-Time Diminished Reality Approach to Privacy in MR Collaboration](https://mlsanity.com/paper/2509.10466/) (2025)

## Abstract (arXiv)

Dynamic objects have a significant impact on the robot's perception of the environment which degrades the performance of essential tasks such as localization and mapping. In this work, we address this problem by synthesizing plausible color, texture and geometry in regions occluded by dynamic objects. We propose the novel geometry-aware DynaFill architecture that follows a coarse-to-fine topology and incorporates our gated recurrent feedback mechanism to adaptively fuse information from previous timesteps. We optimize our architecture using adversarial training to synthesize fine realistic textures which enables it to hallucinate color and depth structure in occluded regions online in a spatially and temporally coherent manner, without relying on future frame information. Casting our inpainting problem as an image-to-image translation task, our model also corrects regions correlated with the presence of dynamic objects in the scene, such as shadows or reflections. We introduce a large-scale hyperrealistic dataset with RGB-D images, semantic segmentation labels, camera poses as well as groundtruth RGB-D information of occluded regions. Extensive quantitative and qualitative evaluations show that our approach achieves state-of-the-art performance, even in challenging weather conditions. Furthermore, we present results for retrieval-based visual localization with the synthesized images that demonstrate the utility of our approach.