# AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation

arXiv: https://arxiv.org/abs/2110.12369 · HTML: https://mlsanity.com/paper/2110.12369/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video](https://mlsanity.com/paper/1807.06667/) (2018)
- [Temporally Distributed Networks for Fast Video Semantic Segmentation](https://mlsanity.com/paper/2004.01800/) (2020)
- [Semantic Video Segmentation by Gated Recurrent Flow Propagation](https://mlsanity.com/paper/1612.08871/) (2016)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [Tent: Fully Test-time Adaptation by Entropy Minimization](https://mlsanity.com/paper/2006.10726/) (2020)
- [Semantic Video CNNs through Representation Warping](https://mlsanity.com/paper/1708.03088/) (2017)
- [Efficient Semantic Video Segmentation with Per-frame Inference](https://mlsanity.com/paper/2002.11433/) (2020)
- [Clockwork Convnets for Video Semantic Segmentation](https://mlsanity.com/paper/1608.03609/) (2016)
- [Temporal Memory Attention for Video Semantic Segmentation](https://mlsanity.com/paper/2102.08643/) (2021)
- [Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation](https://mlsanity.com/paper/2005.10266/) (2020)
- [Real-time Semantic Segmentation with Fast Attention](https://mlsanity.com/paper/2007.03815/) (2020)
- [Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow](https://mlsanity.com/paper/1911.12739/) (2019)
- [Make One-Shot Video Object Segmentation Efficient Again](https://mlsanity.com/paper/2012.01866/) (2020)
- [Joint Optical Flow and Temporally Consistent Semantic Segmentation](https://mlsanity.com/paper/1607.07716/) (2016)
- [STFCN: Spatio-Temporal FCN for Semantic Video Segmentation](https://mlsanity.com/paper/1608.05971/) (2016)
- [Exploiting Temporality for Semi-Supervised Video Segmentation](https://mlsanity.com/paper/1908.11309/) (2019)

## Built on by (most cited first, 30 of 31)

- [Robust Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2303.13899/) (2023)
- [EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization](https://mlsanity.com/paper/2303.01904/) (2023)
- [Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypes](https://mlsanity.com/paper/2207.11707/) (2022)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2410.12790/) (2024)
- [X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detection](https://mlsanity.com/paper/2303.02203/) (2023)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Active Test-Time Adaptation: Theoretical Analyses and An Algorithm](https://mlsanity.com/paper/2404.05094/) (2024)
- [DARTH: Holistic Test-time Adaptation for Multiple Object Tracking](https://mlsanity.com/paper/2310.01926/) (2023)
- [Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation](https://mlsanity.com/paper/2204.05370/) (2022)
- [DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction](https://mlsanity.com/paper/2303.01573/) (2023)
- [GeoAdapt: Self-Supervised Test-Time Adaptation in LiDAR Place Recognition Using Geometric Priors](https://mlsanity.com/paper/2308.04638/) (2023)
- [Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation](https://mlsanity.com/paper/2309.08020/) (2023)
- [TransAdapt: A Transformative Framework for Online Test Time Adaptive Semantic Segmentation](https://mlsanity.com/paper/2302.14611/) (2023)
- [Unified Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2311.13254/) (2023)
- [HGL: Hierarchical Geometry Learning for Test-time Adaptation in 3D Point Cloud Segmentation](https://mlsanity.com/paper/2407.12387/) (2024)
- [Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation](https://mlsanity.com/paper/2307.04470/) (2023)
- [Deep Common Feature Mining for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2403.02689/) (2024)
- [Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2411.01179/) (2024)
- [Towards Open-Vocabulary Video Semantic Segmentation](https://mlsanity.com/paper/2412.09329/) (2024)
- [DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation](https://mlsanity.com/paper/2506.23104/) (2025)
- [Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation](https://mlsanity.com/paper/2604.10950/) (2026)
- [Test-time Adaptation Meets Image Enhancement: Improving Accuracy via Uncertainty-aware Logit Switching](https://mlsanity.com/paper/2403.17423/) (2024)
- [APCoTTA: Continual Test-Time Adaptation for Semantic Segmentation of Airborne LiDAR Point Clouds](https://mlsanity.com/paper/2505.09971/) (2025)
- [MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion](https://mlsanity.com/paper/2405.19921/) (2024)
- [Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptation](https://mlsanity.com/paper/2506.19022/) (2025)
- [X-Align++: cross-modal cross-view alignment for Bird's-eye-view segmentation](https://mlsanity.com/paper/2306.03810/) (2023)
- [HSDA: High-frequency Shuffle Data Augmentation for Bird's-Eye-View Map Segmentation](https://mlsanity.com/paper/2412.06127/) (2024)
- [Uncertainty-aware Test-Time Training (UT3) for Efficient On-the-fly Domain Adaptive Dense Regression](https://mlsanity.com/paper/2509.03012/) (2025)

## Abstract (arXiv)

In video segmentation, generating temporally consistent results across frames is as important as achieving frame-wise accuracy. Existing methods rely either on optical flow regularization or fine-tuning with test data to attain temporal consistency. However, optical flow is not always avail-able and reliable. Besides, it is expensive to compute. Fine-tuning the original model in test time is cost sensitive. This paper presents an efficient, intuitive, and unsupervised online adaptation method, AuxAdapt, for improving the temporal consistency of most neural network models. It does not require optical flow and only takes one pass of the video. Since inconsistency mainly arises from the model's uncertainty in its output, we propose an adaptation scheme where the model learns from its own segmentation decisions as it streams a video, which allows producing more confident and temporally consistent labeling for similarly-looking pixels across frames. For stability and efficiency, we leverage a small auxiliary segmentation network (AuxNet) to assist with this adaptation. More specifically, AuxNet readjusts the decision of the original segmentation network (Main-Net) by adding its own estimations to that of MainNet. At every frame, only AuxNet is updated via back-propagation while keeping MainNet fixed. We extensively evaluate our test-time adaptation approach on standard video benchmarks, including Cityscapes, CamVid, and KITTI. The results demonstrate that our approach provides label-wise accurate, temporally consistent, and computationally efficient adaptation (5+ folds overhead reduction comparing to state-of-the-art test-time adaptation methods).