Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Not on the map. Why?
Abstract (arXiv)
Multimodal fusion emerges as an appealing technique to improve model performances on many tasks. Nevertheless, the robustness of such fusion methods is rarely involved in the present literature. In this paper, we propose a training-free robust late-fusion method by exploiting conditional independence assumption and Jacobian regularization. Our key is to minimize the Frobenius norm of a Jacobian matrix, where the resulting optimization problem is relaxed to a tractable Sylvester equation. Furthermore, we provide a theoretical error bound of our method and some insights about the function of the extra modality. Several numerical experiments on AV-MNIST, RAVDESS, and VGGsound demonstrate the efficacy of our method under both adversarial attacks and random corruptions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 5 Apr 2022.
Builds on
28 of 37 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2018Robustness May Be at Odds with Accuracycited by 880
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
- 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
- 2019Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challengescited by 314
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
- 2020How Does Mixup Help With Robustness and Generalization?cited by 193
- 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
16 more
- 2019Robust Learning with Jacobian Regularizationcited by 129
- 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 124
- 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
- 2020Deep Multimodal Fusion by Channel Exchangingcited by 118
- 2018Improving DNN Robustness to Adversarial Attacks using Jacobian Regularizationcited by 88
- 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
- 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
- 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
- 2019Jacobian Adversarially Regularized Networks for Robustnesscited by 40
- 2017Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environmentscited by 34
- 2017Gradient Regularization Improves Accuracy of Discriminative Modelscited by 26
- 2021Multimodal Knowledge Expansioncited by 26
- 2020TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learningcited by 19
- 2018Robust Deep Multi-modal Learning Based on Gated Information Fusion Networkcited by 18
- 2019On Single Source Robustness in Deep Fusion Modelscited by 12
- 2018Water from Two Rocks: Maximizing the Mutual Informationcited by 10
Built on by
Citing papers per year: 2022: 2.
2 papers.