mlsanityData to 7 Oct 2026

Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model

arXiv 2402.19150 ↗First posted 29 Feb 2024cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Large Vision-Language Models (LVLMs) rely on vision encoders and Large Language Models (LLMs) to exhibit remarkable capabilities on various multi-modal tasks in the joint space of vision and language. However, typographic attacks, which disrupt Vision-Language Models (VLMs) such as Contrastive Language-Image Pretraining (CLIP), have also been expected to be a security threat to LVLMs. Firstly, we verify typographic attacks on current well-known commercial and open-source LVLMs and uncover the widespread existence of this threat. Secondly, to better assess this vulnerability, we propose the most comprehensive and largest-scale Typographic Dataset to date. The Typographic Dataset not only considers the evaluation of typographic attacks under various multi-modal tasks but also evaluates the effects of typographic attacks, influenced by texts generated with diverse factors. Based on the evaluation results, we investigate the causes why typographic attacks impacting VLMs and LVLMs, leading to three highly insightful discoveries. During the process of further validating the rationality of our discoveries, we can reduce the performance degradation caused by typographic attacks from 42.07% to 13.90%. Code and Dataset are available in hrefhttps://github.com/ChaduCheng/TypoDeceptions

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 71 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

57 of 71 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images20 Jan
  2. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  5. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  6. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  7. 2023Defense-Prefix for Preventing Typographic Attacks on CLIP10 Apr
  8. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
37 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  15. 2023A Survey on Multimodal Large Language Modelscited by 898
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  17. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  18. 2021Extract Free Dense Labels from CLIPcited by 580
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  20. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  21. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  22. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  23. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  24. 2023Teaching CLIP to Count to Tencited by 194
  25. 2018TallyQA: Answering Complex Counting Questionscited by 152
  26. 2020Resolution Adaptive Networks for Efficient Inferencecited by 146
  27. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  28. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  29. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  30. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  31. 2021CondenseNet V2: Sparse Feature Reactivation for Deep Networkscited by 34
  32. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  33. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  34. 2023ACT-Diffusion: Efficient Adversarial Consistency Training for One-step Diffusion Modelscited by 5
  35. 2023Improve Video Representation with Temporal Adversarial Augmentationcited by 3
  36. 2023RBFormer: Improve Adversarial Robustness of Transformer by Robust Biascited by 3
  37. 2023Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking Neural Networkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  3. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  4. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  5. 2024Which Model Generated This Image? A Model-Agnostic Approach for Origin Attributioncited by 15
  6. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  7. 2025CHAI: Command Hijacking against embodied AIcited by 9
  8. 2025PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualizationcited by 8
  9. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  10. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 6
  11. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5
  12. 2026SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cuescited by 4
10 more
  1. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  2. 2025AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agentscited by 3
  3. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 3
  4. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  5. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  6. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  7. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Modelscited by 2
  8. 2026Token-Efficient Multimodal Reasoning via Image Prompt Packagingcited by 0
  9. 2026Towards Robustness against Typographic Attack with Training-free Concept Localizationcited by 0
  10. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models
  2. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  3. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models
  4. 2024Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks
  5. 2026Typographic Attack Against VLM-based AI-generated Image Detection
  6. 2026A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning
  7. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
  8. 2026One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations