mlsanityData to 7 Oct 2026

Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment

arXiv 2602.10161 ↗First posted 10 Feb 2026cs.CR cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Omni-modal Large Language Models (OLLMs) greatly expand LLMs' multimodal capabilities but also introduce cross-modal safety risks. However, a systematic understanding of vulnerabilities in omni-modal interactions remains lacking. To bridge this gap, we establish a modality-semantics decoupling principle and construct the AdvBench-Omni dataset, which reveals a significant vulnerability in OLLMs. Mechanistic analysis uncovers a Mid-layer Dissolution phenomenon driven by refusal vector magnitude shrinkage, alongside the existence of a modal-invariant pure refusal direction. Inspired by these insights, we extract a golden refusal vector using Singular Value Decomposition and propose OmniSteer, which utilizes lightweight adapters to modulate intervention intensity adaptively. Extensive experiments show that our method not only increases the Refusal Success Rate against harmful inputs from 69.9% to 91.2%, but also effectively preserves the general capabilities across all modalities. Our code is available at: https://github.com/zhrli324/omni-safety-research.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 57 of its 68 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

57 of 68 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
38 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2016Deep Variational Information Bottleneckcited by 915
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2023Steering Language Models With Activation Engineeringcited by 824
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  8. 2019On Mutual Information Maximization for Representation Learningcited by 315
  9. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  10. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  11. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  12. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  13. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  14. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  15. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  16. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  17. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  18. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  19. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  20. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  21. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  22. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  23. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  24. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  25. 2024A Survey on Speech Large Language Models for Understandingcited by 71
  26. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  27. 2019Semantics Disentangling for Text-to-Image Generationcited by 61
  28. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 57
  29. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  30. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  31. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  32. 2024Safety Alignment for Vision Language Modelscited by 27
  33. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
  34. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  35. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  36. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  37. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  38. 2025SafeVid: Toward Safety Aligned Video Large Multimodal Modelscited by 9

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Codecited by 0
  2. 2026SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMscited by 0
  3. 2026SPARK: Representation-Level KV Memory Alignment for Safer Vision-Language Modelscited by 0
  4. 2026Causal Localization of the Refusal Direction in Audio Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
  2. 2025OmniGuard: Unified Omni-Modal Guardrails with Deliberate Reasoning
  3. 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs
  4. 2026Harnessing Textual Refusal Directions for Multimodal Safety
  5. 2026Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction
  6. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
  7. 2026Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
  8. 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models