# To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentation

arXiv: https://arxiv.org/abs/2307.15063 · HTML: https://mlsanity.com/paper/2307.15063/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 82 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 30 of 63)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2101.10979/) (2021)
- [FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation](https://mlsanity.com/paper/1612.02649/) (2016)
- [DACS: Domain Adaptation via Cross-domain Mixed Sampling](https://mlsanity.com/paper/2007.08702/) (2020)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Progressive Feature Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1811.08585/) (2018)
- [Instance Adaptive Self-Training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2008.12197/) (2020)

## Built on by (most cited first, 14 of 14)

- [Federated Online Adaptation for Deep Stereo](https://mlsanity.com/paper/2405.14873/) (2024)
- [Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2503.21780/) (2025)
- [Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection](https://mlsanity.com/paper/2409.08566/) (2024)
- [Resilient Practical Test-Time Adaptation: Soft Batch Normalization Alignment and Entropy-driven Memory Bank](https://mlsanity.com/paper/2401.14619/) (2024)
- [Test Time Training for Industrial Anomaly Segmentation](https://mlsanity.com/paper/2404.03743/) (2024)
- [DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation](https://mlsanity.com/paper/2506.23104/) (2025)
- [Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation](https://mlsanity.com/paper/2604.10950/) (2026)
- [Improving Online Source-free Domain Adaptation for Object Detection by Unsupervised Data Acquisition](https://mlsanity.com/paper/2310.19258/) (2023)
- [Zero-Shot Anomaly Detection with Dual-Branch Prompt Selection](https://mlsanity.com/paper/2508.00777/) (2025)
- [RobIA: Robust Instance-aware Continual Test-time Adaptation for Deep Stereo](https://mlsanity.com/paper/2511.10107/) (2025)
- [Test-Time Adaptation for Anomaly Segmentation via Topology-Aware Optimal Transport Chaining](https://mlsanity.com/paper/2601.20333/) (2026)
- [MUSDA: Multi-source Multi-modality Unsupervised Domain Adaptive 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2605.10026/) (2026)
- [Learning Topology-Aware Representations via Test-Time Adaptation for Anomaly Segmentation](https://mlsanity.com/paper/2606.28268/) (2026)
- [When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation](https://mlsanity.com/paper/2609.37602/) (2026)

## Abstract (arXiv)

The goal of Online Domain Adaptation for semantic segmentation is to handle unforeseeable domain changes that occur during deployment, like sudden weather events. However, the high computational costs associated with brute-force adaptation make this paradigm unfeasible for real-world applications. In this paper we propose HAMLET, a Hardware-Aware Modular Least Expensive Training framework for real-time domain adaptation. Our approach includes a hardware-aware back-propagation orchestration agent (HAMT) and a dedicated domain-shift detector that enables active control over when and how the model is adapted (LT). Thanks to these advancements, our approach is capable of performing semantic segmentation while simultaneously adapting at more than 29FPS on a single consumer-grade GPU. Our framework's encouraging accuracy and speed trade-off is demonstrated on OnDA and SHIFT benchmarks through experimental results.