# Progressive Cross-modal Knowledge Distillation for Human Action Recognition

arXiv: https://arxiv.org/abs/2208.08090 · HTML: https://mlsanity.com/paper/2208.08090/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 107 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 30 of 48)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [T-GCN: A Temporal Graph ConvolutionalNetwork for Traffic Prediction](https://mlsanity.com/paper/1811.05320/) (2018)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [A Study of Face Obfuscation in ImageNet](https://mlsanity.com/paper/2103.06191/) (2021)
- [Knowledge Distillation via Route Constrained Optimization](https://mlsanity.com/paper/1904.09149/) (2019)
- [MotifNet: a motif-based Graph Convolutional Network for directed graphs](https://mlsanity.com/paper/1802.01572/) (2018)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction](https://mlsanity.com/paper/1910.02212/) (2019)
- [Privacy Preserving Image-Based Localization](https://mlsanity.com/paper/1903.05572/) (2019)

## Built on by (most cited first, 6 of 6)

- [Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2411.11288/) (2024)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines](https://mlsanity.com/paper/2408.14146/) (2024)
- [KEDformer:Knowledge Extraction Seasonal Trend Decomposition for Long-term Sequence Prediction](https://mlsanity.com/paper/2412.05421/) (2024)
- [Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition](https://mlsanity.com/paper/2305.03810/) (2023)
- [GaugeDefect: Detecting Surface Anomalies by Curvature of Feature Transport](https://mlsanity.com/paper/2609.13282/) (2026)

## Abstract (arXiv)

Wearable sensor-based Human Action Recognition (HAR) has achieved remarkable success recently. However, the accuracy performance of wearable sensor-based HAR is still far behind the ones from the visual modalities-based system (i.e., RGB video, skeleton, and depth). Diverse input modalities can provide complementary cues and thus improve the accuracy performance of HAR, but how to take advantage of multi-modal data on wearable sensor-based HAR has rarely been explored. Currently, wearable devices, i.e., smartwatches, can only capture limited kinds of non-visual modality data. This hinders the multi-modal HAR association as it is unable to simultaneously use both visual and non-visual modality data. Another major challenge lies in how to efficiently utilize multimodal data on wearable devices with their limited computation resources. In this work, we propose a novel Progressive Skeleton-to-sensor Knowledge Distillation (PSKD) model which utilizes only time-series data, i.e., accelerometer data, from a smartwatch for solving the wearable sensor-based HAR problem. Specifically, we construct multiple teacher models using data from both teacher (human skeleton sequence) and student (time-series accelerometer data) modalities. In addition, we propose an effective progressive learning scheme to eliminate the performance gap between teacher and student models. We also designed a novel loss function called Adaptive-Confidence Semantic (ACS), to allow the student model to adaptively select either one of the teacher models or the ground-truth label it needs to mimic. To demonstrate the effectiveness of our proposed PSKD method, we conduct extensive experiments on Berkeley-MHAD, UTD-MHAD, and MMAct datasets. The results confirm that the proposed PSKD method has competitive performance compared to the previous mono sensor-based HAR methods.