mlsanityData to 14 Oct 2026

Instruct2Attack: Language-Guided Semantic Adversarial Attacks

arXiv 2311.15551 ↗First posted 27 Nov 2023cs.CV cs.AI cs.CRcited by 16

Not on the map. Why?

Abstract (arXiv)

We propose Instruct2Attack (I2A), a language-guided semantic attack that generates semantically meaningful perturbations according to free-form language instructions. We make use of state-of-the-art latent diffusion models, where we adversarially guide the reverse diffusion process to search for an adversarial latent code conditioned on the input image and text instruction. Compared to existing noise-based and semantic attacks, I2A generates more natural and diverse adversarial examples while providing better controllability and interpretability. We further automate the attack process with GPT-4 to generate diverse image-specific text instructions. We show that I2A can successfully break state-of-the-art deep neural networks even under strong adversarial defenses, and demonstrate great transferability among a variety of network architectures.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 78 of its 89 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

78 of 89 references.

References · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2023GPT-4 Technical Reportcited by 18,396
  9. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  11. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
38 more
  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2022A ConvNet for the 2020scited by 3,579
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  13. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  14. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  15. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  16. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  17. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  19. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  20. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  21. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  22. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  23. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  24. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  25. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  26. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  27. 2022Diffusion Models for Adversarial Purificationcited by 476
  28. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 460
  29. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  30. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  31. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  32. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 220
  33. 2018Spatially Transformed Adversarial Examplescited by 195
  34. 2018Disentangling Adversarial Robustness and Generalizationcited by 172
  35. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  36. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
  37. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  38. 2021Adversarial purification with Score-based generative modelscited by 134

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  2. 2024Instant Adversarial Purification with Adversarial Consistency Distillationcited by 14
  3. 2024A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapsecited by 13
  4. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  5. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 5
  6. 2024Natural Language Induced Adversarial Imagescited by 4
  7. 2024Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Imagescited by 4
  8. 2025RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectorscited by 4
  9. 2025Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systemscited by 4
  10. 2024Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Modelscited by 3
  11. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
  12. 2024MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attackscited by 1
4 more
  1. 2026REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinationscited by 1
  2. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  3. 2026FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectorscited by 0
  4. 2026Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Imagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  2. 2025Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction
  3. 2026Greedy Coordinate Diffusion: Effective and Semantically Coherent Adversarial Attacks via Diffusion Guidance
  4. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities
  5. 2025Semantic Representation Attack against Aligned Large Language Models
  6. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
  7. 2026Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models
  8. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction