mlsanityData to 14 Oct 2026

QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing

arXiv 2608.13119 ↗First posted 13 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 26 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

14 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP28 Aug
  2. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models7 Apr
  3. 2024Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection7 Aug
  4. 2024Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks1 Feb
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 9

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  9. 2019Scene Text Visual Question Answeringcited by 331

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models
  2. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model
  3. 2025Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models
  4. 2026Towards Robustness against Typographic Attack with Training-free Concept Localization
  5. 2026One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations
  6. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  7. 2025Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Models
  8. 2024Reasoning Robustness of LLMs to Adversarial Typographical Errors