# BoTTA: Benchmarking on-device Test Time Adaptation

arXiv: https://arxiv.org/abs/2504.10149 · HTML: https://mlsanity.com/paper/2504.10149/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Towards Stable Test-Time Adaptation in Dynamic Wild World](https://mlsanity.com/paper/2302.12400/) (2023)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Test-Time Training with Masked Autoencoders](https://mlsanity.com/paper/2209.07522/) (2022)
- [NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation](https://mlsanity.com/paper/2208.05117/) (2022)
- [EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization](https://mlsanity.com/paper/2303.01904/) (2023)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)
- [SoTTA: Robust Test-Time Adaptation on Noisy Data Streams](https://mlsanity.com/paper/2310.10074/) (2023)
- [Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalization](https://mlsanity.com/paper/2309.14949/) (2023)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition](https://mlsanity.com/paper/2310.18562/) (2023)
- [Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification](https://mlsanity.com/paper/2307.03133/) (2023)
- [Evaluation of Test-Time Adaptation Under Computational Time Constraints](https://mlsanity.com/paper/2304.04795/) (2023)
- [Fourier Test-time Adaptation with Multi-level Consistency for Robust Classification](https://mlsanity.com/paper/2306.02544/) (2023)
- [ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection](https://mlsanity.com/paper/2312.14227/) (2023)
- [SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity](https://mlsanity.com/paper/2503.20354/) (2025)

## Built on by (most cited first, 4 of 4)

- [NEO: No-Optimization Test-Time Adaptation through Latent Re-Centering](https://mlsanity.com/paper/2510.05635/) (2025)
- [Audio-Visual Continual Test-Time Adaptation without Forgetting](https://mlsanity.com/paper/2602.18528/) (2026)
- [Position Paper: From Edge AI to Adaptive Edge AI](https://mlsanity.com/paper/2604.07360/) (2026)
- [When Adaptation Hurts: Split Sensitivity and Person-Level Negative Transfer in Federated Wearable Onboarding](https://mlsanity.com/paper/2609.27819/) (2026)

## Abstract (arXiv)

The performance of deep learning models depends heavily on test samples at runtime, and shifts from the training data distribution can significantly reduce accuracy. Test-time adaptation (TTA) addresses this by adapting models during inference without requiring labeled test data or access to the original training set. While research has explored TTA from various perspectives like algorithmic complexity, data and class distribution shifts, model architectures, and offline versus continuous learning, constraints specific to mobile and edge devices remain underexplored. We propose BoTTA, a benchmark designed to evaluate TTA methods under practical constraints on mobile and edge devices. Our evaluation targets four key challenges caused by limited resources and usage conditions: (i) limited test samples, (ii) limited exposure to categories, (iii) diverse distribution shifts, and (iv) overlapping shifts within a sample. We assess state-of-the-art TTA methods under these scenarios using benchmark datasets and report system-level metrics on a real testbed. Furthermore, unlike prior work, we align with on-device requirements by advocating periodic adaptation instead of continuous inference-time adaptation. Experiments reveal key insights: many recent TTA algorithms struggle with small datasets, fail to generalize to unseen categories, and depend on the diversity and complexity of distribution shifts. BoTTA also reports device-specific resource use. For example, while SHOT improves accuracy by 2.25× with 512 adaptation samples, it uses 1.08× peak memory on Raspberry Pi versus the base model. BoTTA offers actionable guidance for TTA in real-world, resource-constrained deployments.