mlsanityData to 14 Oct 2026

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

arXiv 2405.17678 ↗First posted 27 May 2024cs.CV cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Achieving zero-shot adversarial robustness without sacrificing generalization remains challenging for foundation models such as CLIP, especially under large adversarial perturbations. Through empirical analyses, we identify three critical yet overlooked issues: (1) Logit margins exhibit a stable offset between small and large adversarial perturbations, suggesting that explicitly adjusting margins could improve robustness against unseen large perturbations. (2) A significant negative correlation exists between logit margin and inter-class semantic similarity, indicating that semantic structures are insufficiently leveraged by existing methods. (3) Existing methods for adjusting text embeddings disrupt the intrinsic semantic consistency established by pre-trained models, undermining generalization capability. Motivated by these findings, we propose a novel Text-Image Mutual Awareness (TIMA) framework, including a Text-Aware Image (TAI) tuning module with an Adaptive Semantic-Aware Margin (ASAM) to explicitly calibrate logit margins, and an Image-Aware Text (IAT) tuning module with Semantic Consistent Minimum Hyperspherical Energy (SC-MHE) to preserve semantic consistency. Comprehensive experiments validate that TIMA significantly outperforms existing approaches by effectively addressing the identified limitations.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

33 of 55 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2016Adversarial examples in the physical worldcited by 1,769
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2016Large-Margin Softmax Loss for Convolutional Neural Networkscited by 296
21 more
  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  2. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  3. 2018MMA Training: Direct Input Space Margin Maximization through Adversarial Trainingcited by 172
  4. 2019Metric Learning for Adversarial Robustnesscited by 77
  5. 2018Learning towards Minimum Hyperspherical Energycited by 75
  6. 2019Adversarial Defense by Restricting the Hidden Space of Deep Neural Networkscited by 56
  7. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  8. 2023Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustnesscited by 36
  9. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  10. 2019Regularizing Neural Networks via Minimizing Hyperspherical Energycited by 25
  11. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  12. 2023Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularizationcited by 21
  13. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalizationcited by 17
  14. 2020MMA Regularization: Decorrelating Weights of Neural Networks by Maximizing the Minimal Anglescited by 15
  15. 2023Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gapcited by 14
  16. 2023Adversarial Attacks on Foundational Vision Modelscited by 13
  17. 2023On the Importance of Backbone to the Adversarial Robustness of Object Detectorscited by 12
  18. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  19. 2021Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learningcited by 8
  20. 2022Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweightingcited by 6
  21. 2022Boosting Adversarial Robustness From The Perspective of Effective Margin Regularizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Balancing Multimodal Domain Generalization via Gradient Modulation and Projectioncited by 1
  2. 2026Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Modelscited by 0