mlsanityData to 7 Oct 2026

Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models

arXiv 2503.11519 ↗First posted 14 Mar 2025cs.CV cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Current Cross-Modality Generation Models (GMs) demonstrate remarkable capabilities in various generative tasks. Given the ubiquity and information richness of vision modality inputs in real-world scenarios, Cross-Vision tasks, encompassing Vision-Language Perception (VLP) and Image-to-Image (I2I), have attracted significant attention. Large Vision Language Models (LVLMs) and I2I Generation Models (GMs) are employed to handle VLP and I2I tasks, respectively. Previous research indicates that printing typographic words into input images significantly induces LVLMs and I2I GMs to produce disruptive outputs that are semantically aligned with those words. Additionally, visual prompts, as a more sophisticated form of typography, are also revealed to pose security risks to various applications of cross-vision tasks. However, the specific characteristics of the threats posed by visual prompts remain underexplored. In this paper, to comprehensively investigate the performance impact induced by Typographic Visual Prompt Injection (TVPI) in various LVLMs and I2I GMs, we propose the Typographic Visual Prompts Injection Dataset and thoroughly evaluate the TVPI security risks on various open-source and closed-source LVLMs and I2I GMs under visual prompts with different target semantics, deepening the understanding of TVPI threats.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

38 of 54 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
26 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  6. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  7. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  8. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  9. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  10. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Modelscited by 154
  11. 2018TallyQA: Answering Complex Counting Questionscited by 152
  12. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  13. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  14. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  15. 2022AT-DDPM: Restoring Faces degraded by Atmospheric Turbulence using Denoising Diffusion Probabilistic Modelscited by 33
  16. 2024FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understandingcited by 26
  17. 2023GCD-DDPM: A Generative Change Detection Model Based on Difference-Feature Guided DDPMcited by 21
  18. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  19. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  20. 2024Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injectioncited by 17
  21. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiadscited by 16
  22. 2023SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric Positive Definite Spacecited by 13
  23. 2025Typographic Attacks in a Multi-Image Settingcited by 8
  24. 2024Cultural Heritage 3D Reconstruction with Diffusion Networkscited by 3
  25. 2025PyPotteryInk: One-Step Diffusion Model for Sketch to Publication-ready Archaeological Drawingscited by 3
  26. 2024Nearly Solved? Robust Deepfake Detection Requires More than Visual Forensicscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  2. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 6
  3. 2026SEF-MAP: Subspace-Decomposed Expert Fusion for Robust Multimodal HD Map Predictioncited by 2
  4. 2026CAMEO: A Conditional and Quality-Aware Multi-Agent Image Editing Orchestratorcited by 2
  5. 2026Extended to Reality: Prompt Injection in 3D Environmentscited by 1
  6. 2025Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrievalcited by 0
  7. 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Modelscited by 0
  8. 2026A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoningcited by 0
  9. 2026Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimizationcited by 0
  10. 2026SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamicscited by 0
  11. 2026Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMscited by 0
  12. 2026Overloading Large Vision-Language Models for Jailbreakingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model
  2. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models
  3. 2026One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations
  4. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models
  5. 2026Towards Robustness against Typographic Attack with Training-free Concept Localization
  6. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
  7. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
  8. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models