# VioLA: Aligning Videos to 2D LiDAR Scans

arXiv: https://arxiv.org/abs/2311.04783 · HTML: https://mlsanity.com/paper/2311.04783/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 30 of 42)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [PREDATOR: Registration of 3D Point Clouds with Low Overlap](https://mlsanity.com/paper/2011.13005/) (2020)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [PointNetLK: Robust & Efficient Point Cloud Registration using PointNet](https://mlsanity.com/paper/1903.05711/) (2019)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [Geometric Transformer for Fast and Robust Point Cloud Registration](https://mlsanity.com/paper/2202.06688/) (2022)
- [RPM-Net: Robust Point Matching using Learned Features](https://mlsanity.com/paper/2003.13479/) (2020)
- [PPFNet: Global Context Aware Local Features for Robust 3D Point Matching](https://mlsanity.com/paper/1802.02669/) (2018)
- [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022)
- [The Perfect Match: 3D Point Cloud Matching with Smoothed Densities](https://mlsanity.com/paper/1811.06879/) (2018)
- [SceneScape: Text-Driven Consistent Scene Generation](https://mlsanity.com/paper/2302.01133/) (2023)
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](https://mlsanity.com/paper/1910.12240/) (2019)
- [ShapeFormer: Transformer-based Shape Completion via Sparse Representation](https://mlsanity.com/paper/2201.10326/) (2022)
- [Robust Point Cloud Registration Framework Based on Deep Graph Matching](https://mlsanity.com/paper/2103.04256/) (2021)
- [Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models](https://mlsanity.com/paper/2207.11514/) (2022)
- [SCPNet: Semantic Scene Completion on Point Cloud](https://mlsanity.com/paper/2303.06884/) (2023)
- [DeepI2P: Image-to-Point Cloud Registration via Deep Classification](https://mlsanity.com/paper/2104.03501/) (2021)
- [Semantic Scene Completion via Integrating Instances and Scene in-the-Loop](https://mlsanity.com/paper/2104.03640/) (2021)
- [View-Guided Point Cloud Completion](https://mlsanity.com/paper/2104.05666/) (2021)
- [CorrI2P: Deep Image-to-Point Cloud Registration via Dense Correspondence](https://mlsanity.com/paper/2207.05483/) (2022)

## Abstract (arXiv)

We study the problem of aligning a video that captures a local portion of an environment to the 2D LiDAR scan of the entire environment. We introduce a method (VioLA) that starts with building a semantic map of the local scene from the image sequence, then extracts points at a fixed height for registering to the LiDAR map. Due to reconstruction errors or partial coverage of the camera scan, the reconstructed semantic map may not contain sufficient information for registration. To address this problem, VioLA makes use of a pre-trained text-to-image inpainting model paired with a depth completion model for filling in the missing scene content in a geometrically consistent fashion to support pose registration. We evaluate VioLA on two real-world RGB-D benchmarks, as well as a self-captured dataset of a large office scene. Notably, our proposed scene completion module improves the pose registration performance by up to 20%.