mlsanityData to 14 Oct 2026

From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defense

arXiv 2601.19448 ↗First posted 27 Jan 2026cs.LG cs.CRcited by 3

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Deep Neural Networks remain inherently vulnerable to backdoor attacks. Traditional test-time defenses largely operate under the paradigm of internal diagnosis methods like model repairing or input robustness, yet these approaches are often fragile under advanced attacks as they remain entangled with the victim model's corrupted parameters. We propose a paradigm shift from Internal Diagnosis to External Semantic Auditing, arguing that effective defense requires decoupling safety from the victim model via an independent, semantically grounded auditor. To this end, we present a framework harnessing Universal Vision-Language Models (VLMs) as evolving semantic gatekeepers. We introduce PRISM (Prototype Refinement & Inspection via Statistical Monitoring), which overcomes the domain gap of general VLMs through two key mechanisms: a Hybrid VLM Teacher that dynamically refines visual prototypes online, and an Adaptive Router powered by statistical margin monitoring to calibrate gating thresholds in real-time. Extensive evaluation across 17 datasets and 11 attack types demonstrates that PRISM achieves state-of-the-art performance, suppressing Attack Success Rate to <1% on CIFAR-10 while improving clean accuracy, establishing a new standard for model-agnostic, externalized security.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 44 of its 75 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

44 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Breaking the Stealth-Potency Trade-off in Clean-Image Backdoors with Generative Trigger Optimization10 Nov
  2. 2025CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation7 Jul
  3. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning20 Nov
  4. 2023Label Poisoning is All You Need29 Oct
  5. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks5 Oct

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  9. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
  10. 2020Input-Aware Dynamic Backdoor Attackcited by 313
  11. 2019Label-Consistent Backdoor Attackscited by 296
  12. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 242
27 more
  1. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 235
  2. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  3. 2017Generative Adversarial Perturbationscited by 153
  4. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 128
  5. 2023SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistencycited by 97
  6. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 86
  7. 2022Data-free Backdoor Removal based on Channel Lipschitznesscited by 79
  8. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  9. 2023Detecting Backdoors During the Inference Stage Based on Corruption Robustness Consistencycited by 46
  10. 2023Black-box Backdoor Defense via Zero-shot Image Purificationcited by 45
  11. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 32
  12. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  13. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 31
  14. 2025REFINE: Inversion-Free Backdoor Defense via Model Reprogrammingcited by 22
  15. 2025Pathology-Aware Prototype Evolution via LLM-Driven Semantic Disambiguation for Multicenter Diabetic Retinopathy Diagnosiscited by 22
  16. 2024BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learningcited by 19
  17. 2022Backdoor Cleansing with Unlabeled Datacited by 17
  18. 2025One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIPcited by 15
  19. 2025From Samples to Scenarios: A New Paradigm for Probabilistic Forecastingcited by 10
  20. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 10
  21. 2025MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representationscited by 8
  22. 2025Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarioscited by 8
  23. 2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
  24. 2024Explanation based Bias Decoupling Regularization for Natural Language Inferencecited by 3
  25. 2025Seal Your Backdoor with Variational Defensecited by 2
  26. 2026Certified Unlearning in Decentralized Federated Learningcited by 2
  27. 2025A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When Agent Automation Becomes Profitable: Quantifying and Insuring Autonomous AI Risk through Trace-Economic Underwritingcited by 3
  2. 2026LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioceptioncited by 3
  3. 2026Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Reviewcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  2. 2025Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense
  3. 2026Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
  4. 2024Exploiting the Vulnerability of Large Language Models via Defense-Aware Architectural Backdoor
  5. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
  6. 2025Robust Anti-Backdoor Instruction Tuning in LVLMs
  7. 2026Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuning
  8. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment