mlsanityData to 7 Oct 2026

Skeletonization-Based Adversarial Perturbations on Large Vision Language Model's Mathematical Text Recognition

arXiv 2601.04752 ↗First posted 8 Jan 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

This work explores the visual capabilities and limitations of foundation models by introducing a novel adversarial attack method utilizing skeletonization to reduce the search space effectively. Our approach specifically targets images containing text, particularly mathematical formula images, which are more challenging due to their LaTeX conversion and intricate structure. We conduct a detailed evaluation of both character and semantic changes between original and adversarially perturbed outputs to provide insights into the models' visual interpretation and reasoning abilities. The effectiveness of our method is further demonstrated through its application to ChatGPT, which shows its practical implications in real-world scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 22 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

12 of 22 references.

References · 12

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  6. 2017One pixel attack for fooling deep neural networkscited by 579
  7. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 459
  8. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  9. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2019Efficient Decision-based Black-box Adversarial Attacks on Face Recognitioncited by 173
  12. 2016Image-to-Markup Generation with Coarse-to-Fine Attentioncited by 79

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  2. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models
  3. 2023Misusing Tools in Large Language Models With Visual Adversarial Examples
  4. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models
  5. 2026HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models
  6. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  7. 2022Why are NLP Models Fumbling at Elementary Math? A Survey of Deep Learning based Word Problem Solvers
  8. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization