mlsanityData to 7 Oct 2026

R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation

arXiv 2412.01154 ↗First posted 2 Dec 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Test-time adaptation (TTA) has emerged as a promising solution to tackle the continual domain shift in machine learning by allowing model parameters to change at test time, via self-supervised learning on unlabeled testing data. At the same time, it unfortunately opens the door to unforeseen vulnerabilities for degradation over time. Through a simple theoretical continual TTA model, we successfully identify a risk in the sampling process of testing data that could easily degrade the performance of a continual TTA model. We name this risk as Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a result of the unintended query from general TTA users. The risk posed by RIP is also highly realistic, as it does not require prior knowledge of model parameters or modification of testing samples. This simple requirement makes RIP as the first black-box TTA attack algorithm that stands out from existing white-box attempts. We extensively benchmark the performance of the most recent continual TTA approaches when facing the RIP attack, providing insights on its success, and laying out potential roadmaps that could enhance the resilience of future continual TTA systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

37 of 57 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  10. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  11. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  12. 2021Domain Generalization: A Surveycited by 760
25 more
  1. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  2. 2022Continual Test-Time Domain Adaptationcited by 520
  3. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  4. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  5. 2019Contrastive Adaptation Network for Unsupervised Domain Adaptationcited by 407
  6. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  7. 2019Simple Black-box Adversarial Attackscited by 249
  8. 2021Improving Robustness using Generated Datacited by 233
  9. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 185
  10. 2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
  11. 2022Robust Mean Teacher for Continual and Gradual Test-Time Adaptationcited by 102
  12. 2021Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformationcited by 67
  13. 2023Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correctioncited by 65
  14. 2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
  15. 2020Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentationcited by 61
  16. 2021Adversarial Example Detection for DNN Models: A Review and Experimental Comparisoncited by 51
  17. 2023RDumb: A simple approach that questions our progress in continual test-time adaptationcited by 50
  18. 2017Detecting Adversarial Image Examples in Deep Networks with Adaptive Noise Reductioncited by 49
  19. 2023Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalizationcited by 34
  20. 2021Grey-box Adversarial Attack And Defence For Sentiment Classificationcited by 28
  21. 2023Persistent Test-time Adaptation in Recurring Testing Scenarioscited by 18
  22. 2023Uncovering Adversarial Risks of Test-Time Adaptationcited by 12
  23. 2024MedBN: Robust Test-Time Adaptation against Malicious Test Samplescited by 10
  24. 2023Test-Time Poisoning Attacks Against Test-Time Adaptation Modelscited by 7
  25. 2023Improving the Robustness of 3D Human Pose Estimation: A Benchmark and Learning from Noisy Inputcited by 4

Built on by

Not cited yet.