# NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night

arXiv: https://arxiv.org/abs/2204.05538 · HTML: https://mlsanity.com/paper/2204.05538/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 70 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation](https://mlsanity.com/paper/1809.09478/) (2018)
- [Unsupervised Intra-domain Adaptation for Semantic Segmentation through Self-Supervision](https://mlsanity.com/paper/2004.07703/) (2020)
- [DLOW: Domain Flow for Adaptation and Generalization](https://mlsanity.com/paper/1812.05418/) (2018)

## Built on by (most cited first, 21 of 21)

- [Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation](https://mlsanity.com/paper/2404.11958/) (2024)
- [2PCNet: Two-Phase Consistency Training for Day-to-Night Unsupervised Domain Adaptive Object Detection](https://mlsanity.com/paper/2303.13853/) (2023)
- [Improving Nighttime Driving-Scene Segmentation via Dual Image-adaptive Learnable Filters](https://mlsanity.com/paper/2207.01331/) (2022)
- [Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation](https://mlsanity.com/paper/2312.01220/) (2023)
- [Disentangle then Parse:Night-time Semantic Segmentation with Illumination Disentanglement](https://mlsanity.com/paper/2307.09362/) (2023)
- [Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation](https://mlsanity.com/paper/2307.08779/) (2023)
- [When Semantic Segmentation Meets Frequency Aliasing](https://mlsanity.com/paper/2403.09065/) (2024)
- [Boosting Night-time Scene Parsing with Learnable Frequency](https://mlsanity.com/paper/2208.14241/) (2022)
- [Recurrent Self-Supervised Video Denoising with Denser Receptive Field](https://mlsanity.com/paper/2308.03608/) (2023)
- [GPS-GLASS: Learning Nighttime Semantic Segmentation Using Daytime Video and GPS data](https://mlsanity.com/paper/2207.13297/) (2022)
- [BFANet: Revisiting 3D Semantic Segmentation with Boundary Feature Analysis](https://mlsanity.com/paper/2503.12539/) (2025)
- [Blind Multimodal Quality Assessment of Low-light Images](https://mlsanity.com/paper/2303.10369/) (2023)
- [Understanding Dark Scenes by Contrasting Multi-Modal Observations](https://mlsanity.com/paper/2308.12320/) (2023)
- [PIG: Prompt Images Guidance for Night-Time Scene Parsing](https://mlsanity.com/paper/2406.10531/) (2024)
- [Spatiotemporal Blind-Spot Network with Calibrated Flow Alignment for Self-Supervised Video Denoising](https://mlsanity.com/paper/2412.11820/) (2024)
- [Not All Pixels Are Equal: Learning Pixel Hardness for Semantic Segmentation](https://mlsanity.com/paper/2305.08462/) (2023)
- [RHRSegNet: Relighting High-Resolution Night-Time Semantic Segmentation](https://mlsanity.com/paper/2407.06016/) (2024)
- [Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology](https://mlsanity.com/paper/2408.14672/) (2024)
- [Improving Panoptic Segmentation for Nighttime or Low-Illumination Urban Driving Scenes](https://mlsanity.com/paper/2306.13725/) (2023)
- [A review of advancements in low-light image enhancement using deep learning](https://mlsanity.com/paper/2505.05759/) (2025)
- [Diffusion-Guided Knowledge Distillation for Weakly-Supervised Low-Light Semantic Segmentation](https://mlsanity.com/paper/2507.07578/) (2025)

## Abstract (arXiv)

The semantic segmentation of nighttime scenes is a challenging problem that is key to impactful applications like self-driving cars. Yet, it has received little attention compared to its daytime counterpart. In this paper, we propose NightLab, a novel nighttime segmentation framework that leverages multiple deep learning models imbued with night-aware features to yield State-of-The-Art (SoTA) performance on multiple night segmentation benchmarks. Notably, NightLab contains models at two levels of granularity, i.e. image and regional, and each level is composed of light adaptation and segmentation modules. Given a nighttime image, the image level model provides an initial segmentation estimate while, in parallel, a hardness detection module identifies regions and their surrounding context that need further analysis. A regional level model focuses on these difficult regions to provide a significantly improved segmentation. All the models in NightLab are trained end-to-end using a set of proposed night-aware losses without handcrafted heuristics. Extensive experiments on the NightCity and BDD100K datasets show NightLab achieves SoTA performance compared to concurrent methods.