# Improving Panoptic Segmentation for Nighttime or Low-Illumination Urban Driving Scenes

arXiv: https://arxiv.org/abs/2306.13725 · HTML: https://mlsanity.com/paper/2306.13725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime](https://mlsanity.com/paper/1810.02575/) (2018)
- [Fully Convolutional Networks for Panoptic Segmentation](https://mlsanity.com/paper/2012.00720/) (2020)
- [DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation](https://mlsanity.com/paper/2104.10834/) (2021)
- [Attention-guided Unified Network for Panoptic Segmentation](https://mlsanity.com/paper/1812.03904/) (2018)
- [DeeperLab: Single-Shot Image Parser](https://mlsanity.com/paper/1902.05093/) (2019)
- [Night-time Scene Parsing with a Large Real Dataset](https://mlsanity.com/paper/2003.06883/) (2020)
- [Learning Instance Occlusion for Panoptic Segmentation](https://mlsanity.com/paper/1906.05896/) (2019)
- [See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion](https://mlsanity.com/paper/1908.05868/) (2019)
- [NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night](https://mlsanity.com/paper/2204.05538/) (2022)
- [Single-shot Path Integrated Panoptic Segmentation](https://mlsanity.com/paper/2012.01632/) (2020)

## Abstract (arXiv)

Autonomous vehicles and driving systems use scene parsing as an essential tool to understand the surrounding environment. Panoptic segmentation is a state-of-the-art technique which proves to be pivotal in this use case. Deep learning-based architectures have been utilized for effective and efficient Panoptic Segmentation in recent times. However, when it comes to adverse conditions like dark scenes with poor illumination or nighttime images, existing methods perform poorly in comparison to daytime images. One of the main factors for poor results is the lack of sufficient and accurately annotated nighttime images for urban driving scenes. In this work, we propose two new methods, first to improve the performance, and second to improve the robustness of panoptic segmentation in nighttime or poor illumination urban driving scenes using a domain translation approach. The proposed approach makes use of CycleGAN (Zhu et al., 2017) to translate daytime images with existing panoptic annotations into nighttime images, which are then utilized to retrain a Panoptic segmentation model to improve performance and robustness under poor illumination and nighttime conditions. In our experiments, Approach-1 demonstrates a significant improvement in the Panoptic segmentation performance on the converted Cityscapes dataset with more than +10% PQ, +12% RQ, +2% SQ, +14% mIoU and +10% AP50 absolute gain. Approach-2 demonstrates improved robustness to varied nighttime driving environments. Both the approaches are supported via comprehensive quantitative and qualitative analysis.