# Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient

arXiv: https://arxiv.org/abs/2308.05681 · HTML: https://mlsanity.com/paper/2308.05681/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 30 of 37)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition](https://mlsanity.com/paper/2010.05599/) (2020)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [ClusterFit: Improving Generalization of Visual Representations](https://mlsanity.com/paper/1912.03330/) (2019)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [A Survey on 3D Skeleton-Based Action Recognition Using Learning Method](https://mlsanity.com/paper/2002.05907/) (2020)
- [Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling](https://mlsanity.com/paper/1908.07214/) (2019)
- [Adversarial Attack on Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1909.06500/) (2019)
- [Practical No-box Adversarial Attacks against DNNs](https://mlsanity.com/paper/2012.02525/) (2020)
- [Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations](https://mlsanity.com/paper/2211.13466/) (2022)

## Built on by (most cited first, 10 of 10)

- [Towards Physical World Backdoor Attacks against Skeleton Action Recognition](https://mlsanity.com/paper/2408.08671/) (2024)
- [TASAR: Transfer-based Attack on Skeletal Action Recognition](https://mlsanity.com/paper/2409.02483/) (2024)
- [Adversarial Attacks on Hidden Tasks in Multi-Task Learning](https://mlsanity.com/paper/2405.15244/) (2024)
- [One-Index Vector Quantization Based Adversarial Attack on Image Classification](https://mlsanity.com/paper/2409.01282/) (2024)
- [NoPain: No-box Point Cloud Attack via Optimal Transport Singular Boundary](https://mlsanity.com/paper/2503.00063/) (2025)
- [Pose-Aware Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2504.05700/) (2025)
- [Unsupervised Cross-Domain 3D Human Pose Estimation via Pseudo-Label-Guided Global Transforms](https://mlsanity.com/paper/2504.12699/) (2025)
- [Lifelong Domain Adaptive 3D Human Pose Estimation](https://mlsanity.com/paper/2512.23860/) (2025)
- [Boosting Adversarial Transferability for Skeleton-based Action Recognition via Exploring the Model Posterior Space](https://mlsanity.com/paper/2407.08572/) (2024)
- [Quality-Preserving Imperceptible Adversarial Attack on Skeleton-based Human Action Recognition](https://mlsanity.com/paper/2606.13022/) (2026)

## Abstract (arXiv)

Recently, methods for skeleton-based human activity recognition have been shown to be vulnerable to adversarial attacks. However, these attack methods require either the full knowledge of the victim (i.e. white-box attacks), access to training data (i.e. transfer-based attacks) or frequent model queries (i.e. black-box attacks). All their requirements are highly restrictive, raising the question of how detrimental the vulnerability is. In this paper, we show that the vulnerability indeed exists. To this end, we consider a new attack task: the attacker has no access to the victim model or the training data or labels, where we coin the term hard no-box attack. Specifically, we first learn a motion manifold where we define an adversarial loss to compute a new gradient for the attack, named skeleton-motion-informed (SMI) gradient. Our gradient contains information of the motion dynamics, which is different from existing gradient-based attack methods that compute the loss gradient assuming each dimension in the data is independent. The SMI gradient can augment many gradient-based attack methods, leading to a new family of no-box attack methods. Extensive evaluation and comparison show that our method imposes a real threat to existing classifiers. They also show that the SMI gradient improves the transferability and imperceptibility of adversarial samples in both no-box and transfer-based black-box settings.