mlsanityData to 14 Oct 2026

SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge

arXiv 2511.16743 ↗First posted 20 Nov 2025cs.CV cs.AI cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Improving the safety of vision-language models like CLIP via fine-tuning often comes at a steep price, causing significant drops in their generalization performance. We find this trade-off stems from rigid alignment strategies that force unsafe concepts toward single, predefined safe targets, disrupting the model's learned semantic structure. To address this, we propose a proximity-aware approach: redirecting unsafe concepts to their semantically closest safe alternatives to minimize representational change. We introduce SaFeR-CLIP, a fine-tuning framework that applies this principle of minimal intervention. SaFeR-CLIP successfully reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods while maintaining robust safety. To support more rigorous evaluation, we also contribute NSFW-Caps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift. Our work shows that respecting the geometry of pretrained representations is key to achieving safety without sacrificing performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 59 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

34 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs25 Feb
  2. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment27 Nov
  3. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  4. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  5. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
17 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2019Natural Adversarial Examplescited by 1,408
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  5. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  6. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  7. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 386
  10. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  11. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  12. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  13. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  14. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  15. 2024The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Modelscited by 23
  16. 2025Who is Responsible? The Data, Models, Users or Regulations? A Comprehensive Survey on Responsible Generative AI for a Sustainable Futurecited by 12
  17. 2025Targeted Forgetting of Image Subgroups in CLIP Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Modelscited by 1
  2. 2026TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviorscited by 0
  3. 2026ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models
  2. 2025Hyperbolic Safety-Aware Vision-Language Models
  3. 2025Safe Vision-Language Models via Unsafe Weights Manipulation
  4. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  5. 2025SIA: Enhancing Safety via Intent Awareness for Vision-Language Models
  6. 2026Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
  7. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
  8. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models