# Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025

arXiv: https://arxiv.org/abs/2506.12430 · HTML: https://mlsanity.com/paper/2506.12430/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 67 of its 74 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Builds on (most cited first, 30 of 67)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Transferable Adversarial Attacks for Image and Video Object Detection](https://mlsanity.com/paper/1811.12641/) (2018)
- [Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt](https://mlsanity.com/paper/2406.04031/) (2024)
- [Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction](https://mlsanity.com/paper/2402.18104/) (2024)
- [A Survey on Trustworthy LLM Agents: Threats and Countermeasures](https://mlsanity.com/paper/2503.09648/) (2025)
- [Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance](https://mlsanity.com/paper/2410.16261/) (2024)
- [VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models](https://mlsanity.com/paper/2402.13851/) (2024)
- [Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detection](https://mlsanity.com/paper/2201.08970/) (2022)
- [SEED-Story: Multimodal Long Story Generation with Large Language Model](https://mlsanity.com/paper/2407.08683/) (2024)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models](https://mlsanity.com/paper/2410.18927/) (2024)
- [Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character](https://mlsanity.com/paper/2405.20773/) (2024)
- [Efficient Adversarial Attacks for Visual Object Tracking](https://mlsanity.com/paper/2008.00217/) (2020)
- [Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2502.11054/) (2025)

## Built on by (most cited first, 12 of 12)

- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Uncovering Security Threats and Architecting Defenses in Autonomous Agents: A Case Study of OpenClaw](https://mlsanity.com/paper/2603.12644/) (2026)
- [SecureWebArena: A Holistic Security Evaluation Benchmark for LVLM-based Web Agents](https://mlsanity.com/paper/2510.10073/) (2025)
- [Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing](https://mlsanity.com/paper/2510.01243/) (2025)
- [MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models](https://mlsanity.com/paper/2506.19257/) (2025)
- [SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling](https://mlsanity.com/paper/2508.09105/) (2025)
- [Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytelling](https://mlsanity.com/paper/2510.15068/) (2025)
- [Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems](https://mlsanity.com/paper/2607.26836/) (2026)
- [SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models](https://mlsanity.com/paper/2509.21400/) (2025)
- [A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models](https://mlsanity.com/paper/2604.04488/) (2026)
- [SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense](https://mlsanity.com/paper/2608.10933/) (2026)
- [TempQ-Jail: Query-Constrained Candidate Ranking for Text-to-Video Jailbreak Attacks](https://mlsanity.com/paper/2609.31032/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have enabled transformative advancements across diverse applications but remain susceptible to safety threats, especially jailbreak attacks that induce harmful outputs. To systematically evaluate and improve their safety, we organized the Adversarial Testing & Large-model Alignment Safety Grand Challenge (ATLAS) 2025}. This technical report presents findings from the competition, which involved 86 teams testing MLLM vulnerabilities via adversarial image-text attacks in two phases: white-box and black-box evaluations. The competition results highlight ongoing challenges in securing MLLMs and provide valuable guidance for developing stronger defense mechanisms. The challenge establishes new benchmarks for MLLM safety evaluation and lays groundwork for advancing safer multimodal AI systems. The code and data for this challenge are openly available at https://github.com/NY1024/ATLAS_Challenge_2025.