# SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments

arXiv: https://arxiv.org/abs/2412.00114 · HTML: https://mlsanity.com/paper/2412.00114/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Patching open-vocabulary models by interpolating weights](https://mlsanity.com/paper/2208.05592/) (2022)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [Towards Adversarial Attack on Vision-Language Pre-training Models](https://mlsanity.com/paper/2206.09391/) (2022)
- [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023)
- [Adversarial Camouflage: Hiding Physical-World Attacks with Natural Styles](https://mlsanity.com/paper/2003.08757/) (2020)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [On the Robustness of Large Multimodal Models Against Image Adversarial Attacks](https://mlsanity.com/paper/2312.03777/) (2023)
- [Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images](https://mlsanity.com/paper/2401.11170/) (2024)
- [Disentangling visual and written concepts in CLIP](https://mlsanity.com/paper/2206.07835/) (2022)
- [Fooling thermal infrared pedestrian detectors in real world using small bulbs](https://mlsanity.com/paper/2101.08154/) (2021)
- [An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models](https://mlsanity.com/paper/2403.09766/) (2024)
- [Infrared Invisible Clothing:Hiding from Infrared Detectors at Multiple Angles in Real World](https://mlsanity.com/paper/2205.05909/) (2022)
- [Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks](https://mlsanity.com/paper/2402.00626/) (2024)
- [Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory](https://mlsanity.com/paper/2403.12445/) (2024)
- [HOTCOLD Block: Fooling Thermal Infrared Detectors with a Novel Wearable Design](https://mlsanity.com/paper/2212.05709/) (2022)
- [Unified Adversarial Patch for Cross-modal Attacks in the Physical World](https://mlsanity.com/paper/2307.07859/) (2023)
- [On the Robustness of Segment Anything](https://mlsanity.com/paper/2305.16220/) (2023)
- [Towards Transferable Targeted 3D Adversarial Attack in the Physical World](https://mlsanity.com/paper/2312.09558/) (2023)
- [Defense-Prefix for Preventing Typographic Attacks on CLIP](https://mlsanity.com/paper/2304.04512/) (2023)

## Built on by (most cited first, 22 of 22)

- [SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models](https://mlsanity.com/paper/2504.04893/) (2025)
- [CHAI: Command Hijacking against embodied AI](https://mlsanity.com/paper/2510.00181/) (2025)
- [Defending LVLMs Against Vision Attacks through Partial-Perception Supervision](https://mlsanity.com/paper/2412.12722/) (2024)
- [VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models](https://mlsanity.com/paper/2602.20999/) (2026)
- [Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions](https://mlsanity.com/paper/2508.05430/) (2025)
- [Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models](https://mlsanity.com/paper/2504.01589/) (2025)
- [AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents](https://mlsanity.com/paper/2510.04257/) (2025)
- [Physical Prompt Injection Attacks on Large Vision-Language Models](https://mlsanity.com/paper/2601.17383/) (2026)
- [Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2603.27918/) (2026)
- [Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models](https://mlsanity.com/paper/2512.11899/) (2025)
- [Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models](https://mlsanity.com/paper/2604.12371/) (2026)
- [MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agents](https://mlsanity.com/paper/2412.08014/) (2024)
- [Extended to Reality: Prompt Injection in 3D Environments](https://mlsanity.com/paper/2602.07104/) (2026)
- [If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems](https://mlsanity.com/paper/2604.19844/) (2026)
- [EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards](https://mlsanity.com/paper/2607.00218/) (2026)
- [AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant T2I Adversarial Patches](https://mlsanity.com/paper/2506.09538/) (2025)
- [One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations](https://mlsanity.com/paper/2604.25102/) (2026)
- [Not What You Asked For: Typographic Attacks in Household Robot Manipulation](https://mlsanity.com/paper/2605.18593/) (2026)
- [AEGIS: A Semantic GAN and Evidential Learning Frameworkfor Robust Adversarial Detection in Vision Sensors](https://mlsanity.com/paper/2606.28416/) (2026)
- [Towards Robustness against Typographic Attack with Training-free Concept Localization](https://mlsanity.com/paper/2607.02494/) (2026)
- [InfraQR: Edge-Placed QR-Inspired Structured Patch Attacks on Infrared Vision-Language Models](https://mlsanity.com/paper/2607.07288/) (2026)
- [On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline](https://mlsanity.com/paper/2607.14974/) (2026)

## Abstract (arXiv)

Large vision-language models (LVLMs) have shown remarkable capabilities in interpreting visual content. While existing works demonstrate these models' vulnerability to deliberately placed adversarial texts, such texts are often easily identifiable as anomalous. In this paper, we present the first approach to generate scene-coherent typographic adversarial attacks that mislead advanced LVLMs while maintaining visual naturalness through the capability of the LLM-based agent. Our approach addresses three critical questions: what adversarial text to generate, where to place it within the scene, and how to integrate it seamlessly. We propose a training-free, multi-modal LLM-driven scene-coherent typographic adversarial planning (SceneTAP) that employs a three-stage process: scene understanding, adversarial planning, and seamless integration. The SceneTAP utilizes chain-of-thought reasoning to comprehend the scene, formulate effective adversarial text, strategically plan its placement, and provide detailed instructions for natural integration within the image. This is followed by a scene-coherent TextDiffuser that executes the attack using a local diffusion mechanism. We extend our method to real-world scenarios by printing and placing generated patches in physical environments, demonstrating its practical implications. Extensive experiments show that our scene-coherent adversarial text successfully misleads state-of-the-art LVLMs, including ChatGPT-4o, even after capturing new images of physical setups. Our evaluations demonstrate a significant increase in attack success rates while maintaining visual naturalness and contextual appropriateness. This work highlights vulnerabilities in current vision-language models to sophisticated, scene-coherent adversarial attacks and provides insights into potential defense mechanisms.