# Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition

arXiv: https://arxiv.org/abs/2112.01849 · HTML: https://mlsanity.com/paper/2112.01849/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 11 of 11)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition](https://mlsanity.com/paper/2009.00210/) (2020)
- [Cycle-Contrast for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2010.14810/) (2020)
- [Deep ConvLSTM with self-attention for human activity decoding using wearables](https://mlsanity.com/paper/2005.00698/) (2020)
- [Multidomain Multimodal Fusion For Human Action Recognition Using Inertial Sensors](https://mlsanity.com/paper/2008.09748/) (2020)

## Built on by (most cited first, 25 of 25)

- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition](https://mlsanity.com/paper/2503.07259/) (2025)
- [Progressive Cross-modal Knowledge Distillation for Human Action Recognition](https://mlsanity.com/paper/2208.08090/) (2022)
- [CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning](https://mlsanity.com/paper/2306.17462/) (2023)
- [TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines](https://mlsanity.com/paper/2408.14146/) (2024)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [One-stage Modality Distillation for Incomplete Multimodal Learning](https://mlsanity.com/paper/2309.08204/) (2023)
- [OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks](https://mlsanity.com/paper/2603.09471/) (2026)
- [Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition](https://mlsanity.com/paper/2211.04331/) (2022)
- [VCD: Visual Causality Discovery for Cross-Modal Question Reasoning](https://mlsanity.com/paper/2304.08083/) (2023)
- [MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU](https://mlsanity.com/paper/2408.08144/) (2024)
- [3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detection](https://mlsanity.com/paper/2407.09020/) (2024)
- [KEDformer:Knowledge Extraction Seasonal Trend Decomposition for Long-term Sequence Prediction](https://mlsanity.com/paper/2412.05421/) (2024)
- [Uncertainty-Aware Cross-Modal Knowledge Distillation with Prototype Learning for Multimodal Brain-Computer Interfaces](https://mlsanity.com/paper/2507.13092/) (2025)
- [Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition](https://mlsanity.com/paper/2305.03810/) (2023)
- [Unified Contrastive Fusion Transformer for Multimodal Human Action Recognition](https://mlsanity.com/paper/2309.05032/) (2023)
- [Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillation](https://mlsanity.com/paper/2410.14700/) (2024)
- [MECKD: Deep Learning-Based Fall Detection in Multilayer Mobile Edge Computing With Knowledge Distillation](https://mlsanity.com/paper/2510.03601/) (2025)
- [Uncertainty-Resilient Multimodal Learning via Consistency-Guided Cross-Modal Transfer](https://mlsanity.com/paper/2511.15741/) (2025)
- [MMA: A Momentum Mamba Architecture for Human Activity Recognition with Inertial Sensors](https://mlsanity.com/paper/2511.21550/) (2025)
- [GaugeDefect: Detecting Surface Anomalies by Curvature of Feature Transport](https://mlsanity.com/paper/2609.13282/) (2026)

## Abstract (arXiv)

Human activity recognition (HAR) based on multi-modal approach has been recently shown to improve the accuracy performance of HAR. However, restricted computational resources associated with wearable devices, i.e., smartwatch, failed to directly support such advanced methods. To tackle this issue, this study introduces an end-to-end Vision-to-Sensor Knowledge Distillation (VSKD) framework. In this VSKD framework, only time-series data, i.e., accelerometer data, is needed from wearable devices during the testing phase. Therefore, this framework will not only reduce the computational demands on edge devices, but also produce a learning model that closely matches the performance of the computational expensive multi-modal approach. In order to retain the local temporal relationship and facilitate visual deep learning models, we first convert time-series data to two-dimensional images by applying the Gramian Angular Field ( GAF) based encoding method. We adopted ResNet18 and multi-scale TRN with BN-Inception as teacher and student network in this study, respectively. A novel loss function, named Distance and Angle-wised Semantic Knowledge loss (DASK), is proposed to mitigate the modality variations between the vision and the sensor domain. Extensive experimental results on UTD-MHAD, MMAct, and Berkeley-MHAD datasets demonstrate the effectiveness and competitiveness of the proposed VSKD model which can deployed on wearable sensors.