mlsanityData to 7 Oct 2026

TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models

arXiv 2308.03906 ↗First posted 7 Aug 2023cs.CVcited by 11

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization). Recent work arXiv:2112.07668 has demonstrated successful backdoor attacks on multimodal models for the Visual Question Answering task. Their dual-key backdoor trigger is split across two modalities (image and text), such that the backdoor is activated if and only if the trigger is present in both modalities. We propose TIJO that defends against dual-key attacks through a joint optimization that reverse-engineers the trigger in both the image and text modalities. This joint optimization is challenging in multimodal models due to the disconnected nature of the visual pipeline which consists of an offline feature extractor, whose output is then fused with the text using a fusion module. The key insight enabling the joint optimization in TIJO is that the trigger inversion needs to be carried out in the object detection box feature space as opposed to the pixel space. We demonstrate the effectiveness of our method on the TrojVQA benchmark, where TIJO improves upon the state-of-the-art unimodal methods from an AUC of 0.6 to 0.92 on multimodal dual-key backdoors. Furthermore, our method also improves upon the unimodal baselines on unimodal backdoors. We present ablation studies and qualitative results to provide insights into our algorithm such as the critical importance of overlaying the inverted feature triggers on all visual features during trigger inversion. The prototype implementation of TIJO is available at https://github.com/SRI-CSL/TIJO.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 44 of its 59 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

44 of 59 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Is Multi-Modal Necessarily Better? Robustness Evaluation of Multi-modal Fake News Detection17 Jun
  2. 2022WeDef: Weakly Supervised Backdoor Defense for Text Classification24 May
  3. 2022Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense11 Feb
  4. 2021Dual-Key Multimodal Backdoors for Visual Question Answering14 Dec
  5. 2021Trigger Hunting with a Topological Prior for Trojan Detection15 Oct
  6. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  7. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models6 Oct
  8. 2021Trojan Signatures in DNN Weights7 Sep
  9. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification7 Mar

Other references · 35

  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  3. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  4. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  5. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  6. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  7. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  8. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  9. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  10. 2020Backdoor Learning: A Surveycited by 412
  11. 2019Hidden Trigger Backdoor Attackscited by 359
  12. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
23 more
  1. 2019Label-Consistent Backdoor Attackscited by 285
  2. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 258
  3. 2019A new Backdoor Attack in CNNs by training set corruption without label poisoningcited by 237
  4. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  5. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  6. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  7. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
  8. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  9. 2019TBT: Targeted Neural Network Attack with Bit Trojancited by 112
  10. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  11. 2020Patch-wise Attack for Fooling Deep Neural Networkcited by 78
  12. 2020DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentationcited by 75
  13. 2019Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networkscited by 66
  14. 2019NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanationscited by 49
  15. 2022BadDet: Backdoor Attacks on Object Detectioncited by 45
  16. 2021DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injectioncited by 42
  17. 2022Dangerous Cloaking: Natural Trigger based Backdoor Attacks on Object Detectors in the Physical Worldcited by 20
  18. 2020Cassandra: Detecting Trojaned Networks from Adversarial Perturbationscited by 15
  19. 2022Imperceptible and Multi-channel Backdoor Attack against Deep Neural Networkscited by 10
  20. 2021FooBaR: Fault Fooling Backdoor Attack on Neural Network Trainingcited by 6
  21. 2020Detecting Trojaned DNNs Using Counterfactual Attributionscited by 4
  22. 2021MISA: Online Defense of Trojaned Models using Misattributionscited by 4
  23. 2022PerD: Perturbation Sensitivity-based Neural Trojan Detection Framework on NLP Applicationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 5.

11 papers.

  1. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  2. 2024A Study of Backdoors in Instruction Fine-tuned Language Modelscited by 11
  3. 2024Adversarial Backdoor Defense in CLIPcited by 10
  4. 2024Test-Time Multimodal Backdoor Detection by Contrastive Promptingcited by 5
  5. 2026CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Modelscited by 3
  6. 2024TrojanEdit: Multimodal Backdoor Attack Against Image Editing Modelcited by 1
  7. 2025Defending against Backdoor Attacks via Module Switchingcited by 0
  8. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  9. 2026CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversioncited by 0
  10. 2026VISTA: Auditing Semantic Divergence in Vision-Language Modelscited by 0
  11. 2026DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priorscited by 0