mlsanityData to 7 Oct 2026

On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

arXiv 2410.04682 ↗First posted 7 Oct 2024cs.LG cs.CVcited by 4

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack order, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods. The source code is available at https://github.com/Gorilla-Lab-SCUT/RTTDP.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 56 of its 68 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

56 of 68 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2016Wide Residual Networkscited by 3,057
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2021Convex optimizationcited by 2,479
38 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 844
  3. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  4. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  5. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  6. 2022Continual Test-Time Domain Adaptationcited by 520
  7. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  8. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  9. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  10. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  11. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  12. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  13. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  14. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  15. 2018Adversarial Attacks and Defences: A Surveycited by 259
  16. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 188
  17. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 185
  18. 2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
  19. 2023EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularizationcited by 123
  20. 2021Adversarial Examples Make Strong Poisonscited by 115
  21. 2020QEBA: Query-Efficient Boundary-Based Blackbox Attackcited by 103
  22. 2022Robust Mean Teacher for Continual and Gradual Test-Time Adaptationcited by 102
  23. 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
  24. 2023Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correctioncited by 65
  25. 2022Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Expertscited by 50
  26. 2017Generative Poisoning Attack Method Against Neural Networkscited by 49
  27. 2022Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clusteringcited by 44
  28. 2022A Probabilistic Framework for Lifelong Test-Time Adaptationcited by 44
  29. 2021If your data distribution shifts, use self-learningcited by 36
  30. 2023On the Robustness of Open-World Test-Time Training: Self-Training with Dynamic Prototype Expansioncited by 34
  31. 2023Towards Real-World Test-Time Adaptation: Tri-Net Self-Training with Balanced Normalizationcited by 34
  32. 2023Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalizationcited by 31
  33. 2021Grey-box Adversarial Attack And Defence For Sentiment Classificationcited by 28
  34. 2024PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Imagescited by 24
  35. 2024Adapting to Distribution Shift by Visual Domain Prompt Generationcited by 23
  36. 2023Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering Regularized Self-Trainingcited by 19
  37. 2019POBA-GA: Perturbation Optimized Black-Box Adversarial Attacks via Genetic Algorithmcited by 14
  38. 2024Distribution Alignment for Fully Test-Time Adaptation with Dynamic Online Data Streamscited by 13

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptationcited by 14
  2. 2025FedPoisonTTP: A Threat Model and Poisoning Attack for Federated Test-Time Personalizationcited by 0
  3. 2026Sample-wise Targeted Adversarial Attacks on Test-time Adaptationcited by 0
  4. 2026Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibrationcited by 0