mlsanityData to 14 Oct 2026

Steal the Patch Size: Adversarially Manipulate Vision-Language Models

arXiv 2607.00174 ↗First posted 30 Jun 2026cs.CV cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

We present a black-box model-stealing attack that recovers private vision-tokenizer configurations of deployed vision-language models (VLMs), including the visual patch size and input preprocessing pipeline. The key idea is a task-level side channel induced by ViT-style patchification: when a synthetic grid image is aligned with the hidden patch grid, boundary cues are erased at tokenization, causing periodic accuracy drop. By sweeping the grid cell size and measuring these collapses, we infer the patch size; by introducing padding and a consistency-check test, we further identify whether preprocessing is dynamic- or fixed-resolution and recover the target resize resolution. Across open-source Qwen-VL variants and proprietary models including GPT and Claude, we reliably recover tokenizer-related parameters. Finally, we show that such leakage enables preprocessing-aware transfer attacks and model-targeted adversarial manipulation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 24 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

21 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models2 May
  2. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  3. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  4. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  6. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  7. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 292
  8. 2023Data Filtering Networkscited by 250
  9. 2018Stealing Hyperparameters in Machine Learningcited by 118
  10. 2020MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimationcited by 99
  11. 2022I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defencescited by 75
  12. 2022Fingerprinting Deep Neural Networks Globally via Universal Adversarial Perturbationscited by 61
4 more
  1. 2019Defending Against Model Stealing Attacks with Adaptive Misinformationcited by 57
  2. 2018Stealing Neural Networks via Timing Side Channelscited by 43
  3. 2020Hermes Attack: Steal DNN Models with Lossless Inference Accuracycited by 41
  4. 2022StolenEncoder: Stealing Pre-trained Encoders in Self-supervised Learningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMscited by 0
  2. 2026Certified Interface Aliases: Exact Collisions in Vision-Language Preprocessing, and When They Existcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
  2. 2025When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models
  3. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulated
  4. 2026Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
  5. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  6. 2025Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models
  7. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
  8. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models