mlsanityData to 7 Oct 2026

Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter

arXiv 2407.19981 ↗First posted 29 Jul 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) have been applied in many computer vision tasks and achieved state-of-the-art (SOTA) performance. However, misclassification will occur when DNNs predict adversarial examples which are created by adding human-imperceptible adversarial noise to natural examples. This limits the application of DNN in security-critical fields. In order to enhance the robustness of models, previous research has primarily focused on the unimodal domain, such as image recognition and video understanding. Although multi-modal learning has achieved advanced performance in various tasks, such as action recognition, research on the robustness of RGB-skeleton action recognition models is scarce. In this paper, we systematically investigate how to improve the robustness of RGB-skeleton action recognition models. We initially conducted empirical analysis on the robustness of different modalities and observed that the skeleton modality is more robust than the RGB modality. Motivated by this observation, we propose the formatwordAttention-based formatwordModality formatwordReweighter (formatwordAMR), which utilizes an attention layer to re-weight the two modalities, enabling the model to learn more robust features. Our AMR is plug-and-play, allowing easy integration with multimodal models. To demonstrate the effectiveness of AMR, we conducted extensive experiments on various datasets. For example, compared to the SOTA methods, AMR exhibits a 43.77% improvement against PGD20 attacks on the NTU-RGB+D 60 dataset. Furthermore, it effectively balances the differences in robustness between different modalities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

36 of 50 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  11. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  12. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
24 more
  1. 2017One pixel attack for fooling deep neural networkscited by 579
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  3. 2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 333
  4. 2021Revisiting Skeleton-based Action Recognitioncited by 256
  5. 2018Adversarial Logit Pairingcited by 224
  6. 2020Geometry-aware Instance-reweighted Adversarial Trainingcited by 198
  7. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 151
  8. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  9. 2018Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregationcited by 106
  10. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
  11. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  12. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  13. 2018Sparse Adversarial Perturbations for Videoscited by 56
  14. 2020Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Rankingcited by 42
  15. 2023rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurementcited by 37
  16. 2024AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectorscited by 29
  17. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  18. 2024GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computingcited by 22
  19. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  20. 2023Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detectioncited by 14
  21. 2024Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledgecited by 13
  22. 2022Analysis and Extensions of Adversarial Training for Video Classificationcited by 11
  23. 2022Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosiscited by 5
  24. 2020Adaptive Adversarial Logits Pairingcited by 3

Built on by

Not cited yet.