# Robust Mean Teacher for Continual and Gradual Test-Time Adaptation

arXiv: https://arxiv.org/abs/2211.13081 · HTML: https://mlsanity.com/paper/2211.13081/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 62 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 45)

- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [DACS: Domain Adaptation via Cross-domain Mixed Sampling](https://mlsanity.com/paper/2007.08702/) (2020)
- [Generate To Adapt: Aligning Domains using Generative Adversarial Networks](https://mlsanity.com/paper/1704.01705/) (2017)
- [Instance Adaptive Self-Training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2008.12197/) (2020)

## Built on by (most cited first, 30 of 104)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correction](https://mlsanity.com/paper/2306.00650/) (2023)
- [ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation](https://mlsanity.com/paper/2306.04344/) (2023)
- [In Search of Lost Online Test-time Adaptation: A Survey](https://mlsanity.com/paper/2310.20199/) (2023)
- [Continual Test-time Domain Adaptation via Dynamic Sample Selection](https://mlsanity.com/paper/2310.03335/) (2023)
- [Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation](https://mlsanity.com/paper/2312.12480/) (2023)
- [Each Test Image Deserves A Specific Prompt: Continual Test-Time Adaptation for 2D Medical Image Segmentation](https://mlsanity.com/paper/2311.18363/) (2023)
- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [Persistent Test-time Adaptation in Recurring Testing Scenarios](https://mlsanity.com/paper/2311.18193/) (2023)
- [Ranked Entropy Minimization for Continual Test-Time Adaptation](https://mlsanity.com/paper/2505.16441/) (2025)
- [What, How, and When Should Object Detectors Update in Continually Changing Test Domains?](https://mlsanity.com/paper/2312.08875/) (2023)
- [Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptation](https://mlsanity.com/paper/2311.05858/) (2023)
- [Backpropagation-free Network for 3D Test-time Adaptation](https://mlsanity.com/paper/2403.18442/) (2024)
- [MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging](https://mlsanity.com/paper/2505.11883/) (2025)
- [AUTO: Adaptive Outlier Optimization for Test-Time OOD Detection](https://mlsanity.com/paper/2303.12267/) (2023)
- [Distribution Alignment for Fully Test-Time Adaptation with Dynamic Online Data Streams](https://mlsanity.com/paper/2407.12128/) (2024)
- [Uncovering Adversarial Risks of Test-Time Adaptation](https://mlsanity.com/paper/2301.12576/) (2023)
- [Calibration-free online test-time adaptation for electroencephalography motor imagery decoding](https://mlsanity.com/paper/2311.18520/) (2023)
- [SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting](https://mlsanity.com/paper/2503.12866/) (2025)
- [A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models](https://mlsanity.com/paper/2405.14977/) (2024)
- [PCoTTA: Continual Test-Time Adaptation for Multi-Task Point Cloud Understanding](https://mlsanity.com/paper/2411.00632/) (2024)
- [textttBATCLIP: Bimodal Online Test-Time Adaptation for CLIP](https://mlsanity.com/paper/2412.02837/) (2024)
- [Monitoring Risks in Test-Time Adaptation](https://mlsanity.com/paper/2507.08721/) (2025)
- [Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2309.13604/) (2023)
- [Unraveling Batch Normalization for Realistic Test-Time Adaptation](https://mlsanity.com/paper/2312.09486/) (2023)
- [Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation](https://mlsanity.com/paper/2405.16486/) (2024)
- [SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training](https://mlsanity.com/paper/2408.08295/) (2024)
- [Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation](https://mlsanity.com/paper/2407.09367/) (2024)
- [AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation](https://mlsanity.com/paper/2309.10109/) (2023)

## Abstract (arXiv)

Since experiencing domain shifts during test-time is inevitable in practice, test-time adaption (TTA) continues to adapt the model after deployment. Recently, the area of continual and gradual test-time adaptation (TTA) emerged. In contrast to standard TTA, continual TTA considers not only a single domain shift, but a sequence of shifts. Gradual TTA further exploits the property that some shifts evolve gradually over time. Since in both settings long test sequences are present, error accumulation needs to be addressed for methods relying on self-training. In this work, we propose and show that in the setting of TTA, the symmetric cross-entropy is better suited as a consistency loss for mean teachers compared to the commonly used cross-entropy. This is justified by our analysis with respect to the (symmetric) cross-entropy's gradient properties. To pull the test feature space closer to the source domain, where the pre-trained model is well posed, contrastive learning is leveraged. Since applications differ in their requirements, we address several settings, including having source data available and the more challenging source-free setting. We demonstrate the effectiveness of our proposed method 'robust mean teacher' (RMT) on the continual and gradual corruption benchmarks CIFAR10C, CIFAR100C, and Imagenet-C. We further consider ImageNet-R and propose a new continual DomainNet-126 benchmark. State-of-the-art results are achieved on all benchmarks.