# Evaluation of Test-Time Adaptation Under Computational Time Constraints

arXiv: https://arxiv.org/abs/2304.04795 · HTML: https://mlsanity.com/paper/2304.04795/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Towards Stable Test-Time Adaptation in Dynamic Wild World](https://mlsanity.com/paper/2302.12400/) (2023)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Revisiting Batch Normalization For Practical Domain Adaptation](https://mlsanity.com/paper/1603.04779/) (2016)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Parameter-free Online Test-time Adaptation](https://mlsanity.com/paper/2201.05718/) (2022)
- [Robust Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2303.13899/) (2023)
- [NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation](https://mlsanity.com/paper/2208.05117/) (2022)
- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data](https://mlsanity.com/paper/2108.09020/) (2021)
- [Evaluating the Adversarial Robustness of Adaptive Test-time Defenses](https://mlsanity.com/paper/2202.13711/) (2022)
- [Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering](https://mlsanity.com/paper/2206.02721/) (2022)
- [ActMAD: Activation Matching to Align Distributions for Test-Time-Training](https://mlsanity.com/paper/2211.12870/) (2022)
- [Combating Adversaries with Anti-Adversaries](https://mlsanity.com/paper/2103.14347/) (2021)
- [Enhancing Adversarial Robustness via Test-time Transformation Ensembling](https://mlsanity.com/paper/2107.14110/) (2021)
- [Robustifying Vision Transformer without Retraining from Scratch by Test-Time Class-Conditional Feature Alignment](https://mlsanity.com/paper/2206.13951/) (2022)

## Built on by (most cited first, 15 of 15)

- [Beyond Model Adaptation at Test Time: A Survey](https://mlsanity.com/paper/2411.03687/) (2024)
- [Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification](https://mlsanity.com/paper/2307.03133/) (2023)
- [Ranked Entropy Minimization for Continual Test-Time Adaptation](https://mlsanity.com/paper/2505.16441/) (2025)
- [Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation](https://mlsanity.com/paper/2505.20704/) (2025)
- [BoTTA: Benchmarking on-device Test Time Adaptation](https://mlsanity.com/paper/2504.10149/) (2025)
- [Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models](https://mlsanity.com/paper/2604.15609/) (2026)
- [Turning Adaptation into Assets: Cross-Domain Bridging for Online Vision-Language Navigation](https://mlsanity.com/paper/2605.23257/) (2026)
- [Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation](https://mlsanity.com/paper/2602.06136/) (2026)
- [Personalizing Retrieval using Joint Embeddings or "the Return of Fluffy"](https://mlsanity.com/paper/2510.05411/) (2025)
- [Fluidity Index: Next-Generation Super-intelligence Benchmarks](https://mlsanity.com/paper/2510.20636/) (2025)
- [SNAP: Low-Latency Test-Time Adaptation with Sparse Updates](https://mlsanity.com/paper/2511.15276/) (2025)
- [Rethinking the Teacher-Student Framework for Test-Time Adaptation](https://mlsanity.com/paper/2609.02507/) (2026)
- [Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation](https://mlsanity.com/paper/2609.20700/) (2026)
- [Rethinking Tabular Foundation Models On Data Streams](https://mlsanity.com/paper/2610.05352/) (2026)
- [Efficient Test-Time Adaptation For Robot Policies](https://mlsanity.com/paper/2610.20318/) (2026)

## Abstract (arXiv)

This paper proposes a novel online evaluation protocol for Test Time Adaptation (TTA) methods, which penalizes slower methods by providing them with fewer samples for adaptation. TTA methods leverage unlabeled data at test time to adapt to distribution shifts. Although many effective methods have been proposed, their impressive performance usually comes at the cost of significantly increased computation budgets. Current evaluation protocols overlook the effect of this extra computation cost, affecting their real-world applicability. To address this issue, we propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream, thereby accounting for the method's adaptation speed. We apply our proposed protocol to benchmark several TTA methods on multiple datasets and scenarios. Extensive experiments show that, when accounting for inference speed, simple and fast approaches can outperform more sophisticated but slower methods. For example, SHOT from 2020, outperforms the state-of-the-art method SAR from 2023 in this setting. Our results reveal the importance of developing practical TTA methods that are both accurate and efficient.