mlsanityData to 14 Oct 2026

VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service

arXiv 2506.15755 ↗First posted 18 Jun 2025cs.CV cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters -- an impractical scenario in ML-as-a-service settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. We hope this research raises the community's awareness about the efficiency robustness of VLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 42 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

25 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images20 Jan
  2. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  3. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  12. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
10 more
  1. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  2. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  3. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  4. 2019Simple Black-box Adversarial Attackscited by 251
  5. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  6. 2018Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examplescited by 99
  7. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
  8. 2022NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Modelscited by 50
  9. 2023Dynamic Transformers Provide a False Sense of Efficiencycited by 14
  10. 2024Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Priorcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 3
  2. 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
  3. 2026Resource Consumption Threats in Large Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  2. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  3. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  4. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  5. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  6. 2025FREE: Fast and Robust Vision Language Models with Early Exits
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models
  8. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs