# DVI: Depth Guided Video Inpainting for Autonomous Driving

arXiv: https://arxiv.org/abs/2007.08854 · HTML: https://mlsanity.com/paper/2007.08854/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [Removing Objects From Neural Radiance Fields](https://mlsanity.com/paper/2212.11966/) (2022)
- [Decoupled Spatial-Temporal Transformer for Video Inpainting](https://mlsanity.com/paper/2104.06637/) (2021)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [OmniInsert: Mask-Free Video Insertion of Any Reference via Diffusion Transformer Models](https://mlsanity.com/paper/2509.17627/) (2025)
- [Semi-Supervised Video Inpainting with Cycle Consistency Constraints](https://mlsanity.com/paper/2208.06807/) (2022)
- [Transformer-based Image and Video Inpainting: Current Challenges and Future Directions](https://mlsanity.com/paper/2407.00226/) (2024)
- [Revealing Disocclusions in Temporal View Synthesis through Infilling Vector Prediction](https://mlsanity.com/paper/2110.08805/) (2021)
- [Depth-Aware Endoscopic Video Inpainting](https://mlsanity.com/paper/2407.02675/) (2024)
- [One-Shot Learning Meets Depth Diffusion in Multi-Object Videos](https://mlsanity.com/paper/2408.16704/) (2024)
- [3D-GIMP: When 3D Gaussian Inpainting Meets PatchMatch](https://mlsanity.com/paper/2607.20789/) (2026)
- [PointManifoldCut: Point-wise Augmentation in the Manifold for Point Clouds](https://mlsanity.com/paper/2109.07324/) (2021)
- [Depth-SIMS: Semi-Parametric Image and Depth Synthesis](https://mlsanity.com/paper/2203.03405/) (2022)
- [Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks](https://mlsanity.com/paper/2308.00231/) (2023)
- [Trusted Video Inpainting Localization via Deep Attentive Noise Learning](https://mlsanity.com/paper/2406.13576/) (2024)

## Abstract (arXiv)

To get clear street-view and photo-realistic simulation in autonomous driving, we present an automatic video inpainting algorithm that can remove traffic agents from videos and synthesize missing regions with the guidance of depth/point cloud. By building a dense 3D map from stitched point clouds, frames within a video are geometrically correlated via this common 3D map. In order to fill a target inpainting area in a frame, it is straightforward to transform pixels from other frames into the current one with correct occlusion. Furthermore, we are able to fuse multiple videos through 3D point cloud registration, making it possible to inpaint a target video with multiple source videos. The motivation is to solve the long-time occlusion problem where an occluded area has never been visible in the entire video. To our knowledge, we are the first to fuse multiple videos for video inpainting. To verify the effectiveness of our approach, we build a large inpainting dataset in the real urban road environment with synchronized images and Lidar data including many challenge scenes, e.g., long time occlusion. The experimental results show that the proposed approach outperforms the state-of-the-art approaches for all the criteria, especially the RMSE (Root Mean Squared Error) has been reduced by about 13%.