E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models

Research endeavors have been made in learning robust Multimodal Large Language Models (MLLMs) against jailbreak attacks. However, existing methods for improving MLLMs' robustness still face critical challenges: ding172 how to efficiently tune massive weight parameters and ding173 how to ensure robustness against attacks across both visual and textual modalities. To this end, we propose an Efficient End-to-end Adversarial Training (E2AT) framework for both visual and textual adversarial attacks. Specifically, for the visual aspect, E2AT incorporates an efficient projector-based AT module that aligns the attack samples at the feature level. For training objectives, we propose a Dynamic Joint Multimodal Optimization (DJMO) strategy to enhance generalization ability against jailbreak attacks by dynamically adjusting weights between normal and adversarial objectives. Extensive experiments are conducted with five major jailbreak attack methods across three mainstream MLLMs. Results demonstrate that our E2AT achieves the state-of-the-art performance, outperforming existing baselines by an average margin of 34% across text and image modalities, while maintaining clean task performance. Furthermore, evaluations of real-world embodied intelligent systems highlight the practical applicability of E2AT, paving the way for the development of more secure and reliable multimodal systems. Our code is available on hrefhttps://anonymous.4open.science/r/E2AT568textcolorredhttps://anonymous.4open.science/r/E2AT568.