mlsanityData to 14 Oct 2026

BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

arXiv 2606.02947 ↗First posted 1 Jun 2026cs.LG cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks. Recent work has shown that this paradigm is highly vulnerable to backdoor attacks, and that existing defenses are ineffective in open-ended generation settings. In response, we propose BYORn, a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible given the corresponding image-text inputs and a pretrained model. BYORn identifies such misaligned responses and dynamically replaces them with alternative responses generated by the model, thereby breaking the correlation between triggers and target outputs. The resulting objective gradient corresponds to the gradient of the empirical estimate of the population risk upper bound over the clean data distribution. Empirically, BYORn consistently improves robustness to backdoor attacks while preserving clean-task performance, establishing a new trade-off frontier between generalization and attack success rate. Finally, we demonstrate that BYORn remains effective against adaptive attacks specifically designed to circumvent the proposed defense.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 79 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

63 of 79 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs5 Jun
  2. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuning22 May
  3. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models25 Feb
  4. 2024Backdooring Vision-Language Models with Out-Of-Distribution Data2 Oct
  5. 2024TrojVLM: Backdoor Attack Against Vision Language Models28 Sep
  6. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  7. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  8. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models5 Feb
  9. 2023Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objective25 Nov
  10. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks13 Mar

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2022Visual Prompt Tuningcited by 1,681
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  8. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  13. 2020Backdoor Learning: A Surveycited by 426
  14. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 353
  15. 2019Label-Consistent Backdoor Attackscited by 296
  16. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  18. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  19. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 263
  20. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  21. 2017Neural Trojanscited by 153
  22. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  23. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  24. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  25. 2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 64
  26. 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 63
  27. 2018Learning to Evaluate Image Captioningcited by 53
  28. 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
  29. 2021Dual-Key Multimodal Backdoors for Visual Question Answeringcited by 37
  30. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 33
  31. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Studycited by 28
  32. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  33. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  34. 2023Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attackscited by 19
  35. 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
  36. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  37. 2024BadCM: Invisible Backdoor Attack Against Cross-Modal Learningcited by 9
  38. 2024Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoorscited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
  2. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  3. 2026A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
  4. 2025MTAttack: Multi-Target Backdoor Attacks against Large Vision-Language Models
  5. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment
  6. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models
  7. 2024Exploiting the Vulnerability of Large Language Models via Defense-Aware Architectural Backdoor
  8. 2026EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy