A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has driven major gains in reasoning, perception, and generation across language and vision, yet whether these advances translate into comparable improvements in safety remains unclear, partly due to fragmented evaluations that focus on isolated modalities or threat models. In this report, we present an integrated safety evaluation of six frontier models--GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5--assessing each across language, vision-language, and image generation using a unified protocol that combines benchmark, adversarial, multilingual, and compliance evaluations. By aggregating results into safety leaderboards and model profiles, we reveal a highly uneven safety landscape: while GPT-5.2 demonstrates consistently strong and balanced performance, other models exhibit clear trade-offs across benchmark safety, adversarial robustness, multilingual generalization, and regulatory compliance. Despite strong results under standard benchmarks, all models remain highly vulnerable under adversarial testing, with worst-case safety rates dropping below 6%. Text-to-image models show slightly stronger alignment in regulated visual risk categories, yet remain fragile when faced with adversarial or semantically ambiguous prompts. Overall, these findings highlight that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation design--underscoring the need for standardized, holistic safety assessments to better reflect real-world risk and guide responsible deployment.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 15 Jan 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defensescited by 7
- 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
- 2026Constitutional On-Policy Safe Distillationcited by 1
- 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequencescited by 0
- 2026Visual Distraction Undermines Moral Reasoning in Vision-Language Modelscited by 0
- 2026DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosiscited by 0
- 2026Revisiting the shutdown problemcited by 0
- 2026PanDent: Toward Comprehensive Tooth-Level Structure-Language Consistency in Dental Radiologycited by 0
- 2026PanDent: Toward Comprehensive Tooth-Level Structure-Language Consistency in Dental Radiology29 Jul 2026
- 2026Revisiting the shutdown problem6 Jun 2026
- 2026Constitutional On-Policy Safe Distillation2 Jun 2026
- 2026An Independent Safety Evaluation of Kimi K2.53 Apr 2026
- 2026DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosis30 Mar 2026
- 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses28 Mar 2026
- 2026Visual Distraction Undermines Moral Reasoning in Vision-Language Models17 Mar 2026
- 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences10 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
- 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
- 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
- 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
- 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
- 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
- 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
- 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models