mlsanityData to 14 Oct 2026

A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5

arXiv 2601.10527 ↗First posted 15 Jan 2026cs.AI cs.CL cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has driven major gains in reasoning, perception, and generation across language and vision, yet whether these advances translate into comparable improvements in safety remains unclear, partly due to fragmented evaluations that focus on isolated modalities or threat models. In this report, we present an integrated safety evaluation of six frontier models--GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5--assessing each across language, vision-language, and image generation using a unified protocol that combines benchmark, adversarial, multilingual, and compliance evaluations. By aggregating results into safety leaderboards and model profiles, we reveal a highly uneven safety landscape: while GPT-5.2 demonstrates consistently strong and balanced performance, other models exhibit clear trade-offs across benchmark safety, adversarial robustness, multilingual generalization, and regulatory compliance. Despite strong results under standard benchmarks, all models remain highly vulnerable under adversarial testing, with worst-case safety rates dropping below 6%. Text-to-image models show slightly stronger alignment in regulated visual risk categories, yet remain fragile when faced with adversarial or semantically ambiguous prompts. Overall, these findings highlight that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation design--underscoring the need for standardized, holistic safety assessments to better reflect real-world risk and guide responsible deployment.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. First posted on arXiv 15 Jan 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
  2. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  3. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
  5. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models
  6. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
  7. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  8. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models