mlsanityData to 7 Oct 2026

A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models

arXiv 2603.26984 ↗First posted 27 Mar 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about their reliability in real-world use. While adversarial training has become the leading paradigm for building models that are robust to adversarial attacks, Test-Time Transformations (TTT) have emerged as a promising strategy to boost robustness at inference. In light of this, we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples. Our method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions. We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Question Answering. Code is available at github.com/OmnAI-Lab/Energy-Guided-Test-Time-Defense .

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 81 of its 104 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

81 of 104 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  2. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  3. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  4. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  5. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  6. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  7. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 74

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2019Towards VQA Models That Can Readcited by 1,696
38 more
  1. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  2. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  3. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  4. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  5. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  10. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  11. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  13. 2022Diffusion Models for Adversarial Purificationcited by 469
  14. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  15. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  16. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  17. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  18. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  19. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  20. 2023Are aligned neural networks adversarially aligned?cited by 305
  21. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 243
  22. 2022Test-Time Training with Masked Autoencoderscited by 208
  23. 2016A Theory of Generative ConvNetcited by 174
  24. 2018Disentangling Adversarial Robustness and Generalizationcited by 171
  25. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  26. 2021Adversarial purification with Score-based generative modelscited by 132
  27. 2022Guided Diffusion Model for Adversarial Purificationcited by 90
  28. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  29. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  30. 2018Adversarial examples from computational constraintscited by 79
  31. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  32. 2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
  33. 2020Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Modelscited by 58
  34. 2021The Dimpled Manifold Model of Adversarial Examples in Machine Learningcited by 41
  35. 2020Data-Dependent Randomized Smoothingcited by 37
  36. 2019A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distancecited by 32
  37. 2021Attacking Adversarial Attacks as A Defensecited by 31
  38. 2021Combating Adversaries with Anti-Adversariescited by 27

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Structure-Guided Visual Perturbation Neutralization for LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adversarial Defense in Vision-Language Models: An Overview
  2. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models
  3. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
  4. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  5. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
  6. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
  7. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  8. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models