# MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation

arXiv: https://arxiv.org/abs/2210.07577 · HTML: https://mlsanity.com/paper/2210.07577/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics](https://mlsanity.com/paper/1906.05717/) (2019)
- [Distilled Semantics for Comprehensive Scene Understanding from Videos](https://mlsanity.com/paper/2003.14030/) (2020)
- [Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation](https://mlsanity.com/paper/2005.10266/) (2020)
- [Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks](https://mlsanity.com/paper/1604.07480/) (2016)
- [Geometry meets semantics for semi-supervised monocular depth estimation](https://mlsanity.com/paper/1810.04093/) (2018)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Time-Space Transformers for Video Panoptic Segmentation](https://mlsanity.com/paper/2210.03546/) (2022)

## Built on by (most cited first, 7 of 7)

- [MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion](https://mlsanity.com/paper/2405.19921/) (2024)
- [Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation](https://mlsanity.com/paper/2412.07966/) (2024)
- [Towards Sharper Object Boundaries in Self-Supervised Depth Estimation](https://mlsanity.com/paper/2509.15987/) (2025)
- [Unified Perception: Efficient Depth-Aware Video Panoptic Segmentation with Minimal Annotation Costs](https://mlsanity.com/paper/2303.01991/) (2023)
- [MGNiceNet: Unified Monocular Geometric Scene Understanding](https://mlsanity.com/paper/2411.11466/) (2024)
- [DSER: Spectral Epipolar Representation for Efficient Light Field Depth Estimation](https://mlsanity.com/paper/2508.08900/) (2025)
- [DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2607.26165/) (2026)

## Abstract (arXiv)

Depth-aware video panoptic segmentation tackles the inverse projection problem of restoring panoptic 3D point clouds from video sequences, where the 3D points are augmented with semantic classes and temporally consistent instance identifiers. We propose a novel solution with a multi-task network that performs monocular depth estimation and video panoptic segmentation. Since acquiring ground truth labels for both depth and image segmentation has a relatively large cost, we leverage the power of unlabeled video sequences with self-supervised monocular depth estimation and semi-supervised learning from pseudo-labels for video panoptic segmentation. To further improve the depth prediction, we introduce panoptic-guided depth losses and a novel panoptic masking scheme for moving objects to avoid corrupting the training signal. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that our model with the proposed improvements achieves competitive results and fast inference speed.