mlsanityData to 14 Oct 2026

Jailbreaking Large Vision Language Models in Intelligent Transportation Systems

arXiv 2511.13892 ↗First posted 17 Nov 2025cs.AIcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Large Vision Language Models (LVLMs) demonstrate strong capabilities in multimodal reasoning and many real-world applications, such as visual question answering. However, LVLMs are highly vulnerable to jailbreaking attacks. This paper systematically analyzes the vulnerabilities of LVLMs integrated in Intelligent Transportation Systems (ITS) under carefully crafted jailbreaking attacks. First, we carefully construct a dataset with harmful queries relevant to transportation, following OpenAI's prohibited categories to which the LVLMs should not respond. Second, we introduce a novel jailbreaking attack that exploits the vulnerabilities of LVLMs through image typography manipulation and multi-turn prompting. Third, we propose a multi-layered response filtering defense technique to prevent the model from generating inappropriate responses. We perform extensive experiments with the proposed attack and defense on the state-of-the-art LVLMs (both open-source and closed-source). To evaluate the attack method and defense technique, we use GPT-4's judgment to determine the toxicity score of the generated responses, as well as manual verification. Further, we compare our proposed jailbreaking method with existing jailbreaking techniques and highlight severe security risks involved with jailbreaking attacks with image typography manipulation and multi-turn prompting in the LVLMs integrated in ITS.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 33 of its 45 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

33 of 45 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  11. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  12. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
21 more
  1. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  4. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  5. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  6. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  7. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 80
  8. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 77
  9. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
  10. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 51
  11. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 36
  12. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 33
  13. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  14. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  15. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  16. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  17. 2025Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehiclescited by 9
  18. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  19. 2025System Prompt Extraction Attacks and Defenses in Large Language Modelscited by 6
  20. 2025CEQuest: Benchmarking Large Language Models for Construction Estimationcited by 6
  21. 2025Mapillary Vistas Validation for Fine-Grained Traffic Signs: A Benchmark Revealing Vision-Language Model Limitationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DIVA: Exploiting Cross-Step Conditional Propagation for Visual Jailbreaks in Discrete Diffusion Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  2. 2026Overloading Large Vision-Language Models for Jailbreaking
  3. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  4. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
  5. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image
  6. 2025Enhanced MLLM Black-Box Jailbreaking Attacks and Defenses
  7. 2026Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models
  8. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation