# PVO: Panoptic Visual Odometry

arXiv: https://arxiv.org/abs/2207.01610 · HTML: https://mlsanity.com/paper/2207.01610/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 30 of 44)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects](https://mlsanity.com/paper/1804.09194/) (2018)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [Fully Convolutional Networks for Panoptic Segmentation](https://mlsanity.com/paper/2012.00720/) (2020)
- [SfM-Net: Learning of Structure and Motion from Video](https://mlsanity.com/paper/1704.07804/) (2017)
- [STEP: Segmenting and Tracking Every Pixel](https://mlsanity.com/paper/2102.11859/) (2021)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)

## Built on by (most cited first, 26 of 26)

- [PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction](https://mlsanity.com/paper/2406.06521/) (2024)
- [NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild](https://mlsanity.com/paper/2405.18715/) (2024)
- [Deep Patch Visual SLAM](https://mlsanity.com/paper/2408.01654/) (2024)
- [LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry](https://mlsanity.com/paper/2401.01887/) (2024)
- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion](https://mlsanity.com/paper/2410.24203/) (2024)
- [CityGS-X: A Scalable Architecture for Efficient and Geometrically Accurate Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2503.23044/) (2025)
- [iDF-SLAM: End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking](https://mlsanity.com/paper/2209.07919/) (2022)
- [GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding](https://mlsanity.com/paper/2311.11863/) (2023)
- [StreetSurfGS: Scalable Urban Street Surface Reconstruction with Planar-based Gaussian Splatting](https://mlsanity.com/paper/2410.04354/) (2024)
- [NeuRodin: A Two-stage Framework for High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2408.10178/) (2024)
- [Reinforcement Learning Meets Visual Odometry](https://mlsanity.com/paper/2407.15626/) (2024)
- [ND-SDF: Learning Normal Deflection Fields for High-Fidelity Indoor Reconstruction](https://mlsanity.com/paper/2408.12598/) (2024)
- [D3FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance](https://mlsanity.com/paper/2207.08794/) (2022)
- [DVI-SLAM: A Dual Visual Inertial SLAM Network](https://mlsanity.com/paper/2309.13814/) (2023)
- [RD-VIO: Robust Visual-Inertial Odometry for Mobile Augmented Reality in Dynamic Environments](https://mlsanity.com/paper/2310.15072/) (2023)
- [Large-Scale Gaussian Splatting SLAM](https://mlsanity.com/paper/2505.09915/) (2025)
- [MC-NeRF: Multi-Camera Neural Radiance Fields for Multi-Camera Image Acquisition Systems](https://mlsanity.com/paper/2309.07846/) (2023)
- [DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild](https://mlsanity.com/paper/2411.13291/) (2024)
- [ZeroVO: Visual Odometry with Minimal Assumptions](https://mlsanity.com/paper/2506.08005/) (2025)
- [NGD-SLAM: Towards Real-Time Dynamic SLAM without GPU](https://mlsanity.com/paper/2405.07392/) (2024)
- [Self-Supervised Geometry-Guided Initialization for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2406.00929/) (2024)
- [MVOFormer: Flow-Semantic Transformer for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2606.16474/) (2026)
- [An Earth Rover dataset recorded at the ICRA@40 party](https://mlsanity.com/paper/2407.05735/) (2024)
- [STAG-VIO: Stabilized Prompt-to-Geometry Interface for Robust Dynamic Visual--Inertial Odometry](https://mlsanity.com/paper/2411.19289/) (2024)
- [SPORTS: Simultaneous Panoptic Odometry, Rendering, Tracking and Segmentation for Urban Scenes Understanding](https://mlsanity.com/paper/2510.12749/) (2025)

## Abstract (arXiv)

We present PVO, a novel panoptic visual odometry framework to achieve more comprehensive modeling of the scene motion, geometry, and panoptic segmentation information. Our PVO models visual odometry (VO) and video panoptic segmentation (VPS) in a unified view, which makes the two tasks mutually beneficial. Specifically, we introduce a panoptic update module into the VO Module with the guidance of image panoptic segmentation. This Panoptic-Enhanced VO Module can alleviate the impact of dynamic objects in the camera pose estimation with a panoptic-aware dynamic mask. On the other hand, the VO-Enhanced VPS Module also improves the segmentation accuracy by fusing the panoptic segmentation result of the current frame on the fly to the adjacent frames, using geometric information such as camera pose, depth, and optical flow obtained from the VO Module. These two modules contribute to each other through recurrent iterative optimization. Extensive experiments demonstrate that PVO outperforms state-of-the-art methods in both visual odometry and video panoptic segmentation tasks.