mlsanityData to 7 Oct 2026

Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models

arXiv 2601.22737 ↗First posted 30 Jan 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The robust safety of Vision-Language Large Models (VLLMs) against joint multilingual and multimodal threats remains severely underexplored. Current benchmarks typically isolate these dimensions, being either multilingual but text-only, or multimodal but monolingual. While recent red-teaming efforts attempt to bridge this gap by rendering harmful prompts as images, their overreliance on typography-style visuals and lack of semantically grounded image-text pairs fail to capture realistic cross-modal interactions under multilingual and multimodal conditions. To address this, we introduce Lingua-SafetyBench, a comprehensive benchmark of 100,440 harmful image-text pairs spanning 10 languages. Crucially, Lingua-SafetyBench explicitly partitions data into image-dominant and text-dominant subsets to precisely disentangle sources of risk. Extensive evaluations reveal that current VLLMs retain non-negligible vulnerabilities under these joint inputs. Linguistically, requests in Non-High-Resource Languages (Non-HRLs) and non-Latin scripts generally pose greater threats. Furthermore, analyzing modality-language interactions uncovers a striking asymmetry: in High-Resource Languages (HRLs), models are most vulnerable to image-dominant risks, whereas in Non-HRLs, text-dominant risks severely degrade safety performance. Finally, a controlled study on the Qwen series demonstrates that while model scaling and iterative upgrades improve overall safety, they disproportionately benefit HRLs. This exacerbates the safety disparity between HRLs and Non-HRLs under text-dominant risks, highlighting that achieving robust safety requires dedicated language- and modality-aware alignment strategies beyond mere scaling. The code and dataset will be available at https://github.com/zsxr15/Lingua-SafetyBench.Warning: this paper contains examples with unsafe content.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 66 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

55 of 66 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs16 Feb
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations14 Feb
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety2 Feb
  4. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  5. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends10 Jul
  6. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models11 Jun
  7. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  8. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  9. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2025OpenAI GPT-5 System Cardcited by 910
  9. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  10. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  11. 2023Using an LLM to Help With Code Understandingcited by 222
  12. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
34 more
  1. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  2. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  3. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  4. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  5. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  6. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  8. 2025Qwen3Guard Technical Reportcited by 125
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  11. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  12. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  13. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  14. 2025International AI Safety Reportcited by 64
  15. 2025Safety in Large Reasoning Models: A Surveycited by 58
  16. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  17. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  18. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 35
  19. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  20. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  21. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  22. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  23. 2024Learning to Refuse: Towards Mitigating Privacy Risks in LLMscited by 22
  24. 2025A Survey on Efficient Vision-Language Modelscited by 19
  25. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  26. 2025Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Searchcited by 15
  27. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 15
  28. 2025AI Safety for Everyonecited by 14
  29. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  30. 2026BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearningcited by 8
  31. 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
  32. 2025Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Modelscited by 2
  33. 2026Guardrails for trust, safety, and ethical development and deployment of Large Language Models (LLM)cited by 2
  34. 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
  2. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
  3. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
  4. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  5. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  6. 2025VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration
  7. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study
  8. 2025ELITE: Enhanced Language-Image Toxicity Evaluation for Safety