Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
also in 1 more
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) are increasingly deployed in high-stakes domains where fairness is a critical safety requirement. In practice, these models are continually updated through continual learning (CL) to adapt to evolving tasks and data distributions. Prior work has shown that backdoor attacks can manipulate MLLM responses through hidden triggers, but naively implanted backdoors degrade as models undergo subsequent updates of CL. Although fairness has emerged as a central concern for MLLM deployment, whether backdoor-induced fairness violations can survive CL remains unexplored, leaving two critical questions unanswered: (1) whether a backdoor can reliably induce fairness violations in MLLMs, and (2) whether such fairness-targeted backdoors can persist through continual learning. We bridge this gap by proposing Persistent Fairness Backdoor Attack (PFBA) to inject persistent and group-specific discrimination into MLLMs. Specifically, PFBA achieves this through two novel mechanisms. The Latent Space Fairness Reinforcement reshapes the model's deep feature geometry by anchoring privileged-group representations to preserve utility while repelling and clustering targeted-group representations to sustain discrimination, and the Continual Learning Simulation iteratively optimizes the trigger against simulated parameter drift to ensure backdoor persistence across future updates. Extensive experiments demonstrate that PFBA induces severe fairness disparities that persist across continual learning rounds, evading standard backdoor defenses. The data and code are publicly available at https://github.com/lyygua/PFBA.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 48 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
22 of 48 references.
Closest ancestors · 1
Same direction, newest first
Other references · 21
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2014Deep Learning Face Attributes in the Wildcited by 3,693
- 2016Learning without Forgettingcited by 2,496
- 2018Efficient Lifelong Learning with A-GEMcited by 1,058
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
- 2018Spectral Signatures in Backdoor Attackscited by 484
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
- 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
- 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
- 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 76
- 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
9 more
- 2023CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learningcited by 63
- 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 39
- 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Setscited by 27
- 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
- 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
- 2024Persistent Backdoor Attacks in Continual Learningcited by 13
- 2021Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Modelscited by 9
- 2023BrainWash: A Poisoning Attack to Forget in Continual Learningcited by 9
- 2024BadFair: Backdoored Fairness Attacks with Group-conditioned Triggerscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
- 2024Neutralizing Backdoors through Information Conflicts for Large Language Models
- 2025Robust Anti-Backdoor Instruction Tuning in LVLMs
- 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
- 2026Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs
- 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs
- 2023Beyond Detection: Unveiling Fairness Vulnerabilities in Abusive Language Models
- 2025Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs