# A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation

arXiv: https://arxiv.org/abs/2107.07331 · HTML: https://mlsanity.com/paper/2107.07331/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 72 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://mlsanity.com/paper/2004.01461/) (2020)
- [Blockwisely Supervised Neural Architecture Search with Knowledge Distillation](https://mlsanity.com/paper/1911.13053/) (2019)
- [Learning Student Networks via Feature Embedding](https://mlsanity.com/paper/1812.06597/) (2018)
- [Conditional Teacher-Student Learning](https://mlsanity.com/paper/1904.12399/) (2019)
- [EmbraceNet: A robust deep learning architecture for multimodal classification](https://mlsanity.com/paper/1904.09078/) (2019)
- [Search to Distill: Pearls are Everywhere but not the Eyes](https://mlsanity.com/paper/1911.09074/) (2019)
- [E-PUR: An Energy-Efficient Processing Unit for Recurrent Neural Networks](https://mlsanity.com/paper/1711.07480/) (2017)
- [SparseSense: Human Activity Recognition from Highly Sparse Sensor Data-streams Using Set-based Neural Networks](https://mlsanity.com/paper/1906.02399/) (2019)
- [EmbraceNet for Activity: A Deep Multimodal Fusion Architecture for Activity Recognition](https://mlsanity.com/paper/2004.13918/) (2020)
- [A Framework of Combining Short-Term Spatial/Frequency Feature Extraction and Long-Term IndRNN for Activity Recognition](https://mlsanity.com/paper/2011.00395/) (2020)

## Abstract (arXiv)

Inertial sensor-based human activity recognition (HAR) is the base of many human-centered mobile applications. Deep learning-based fine-grained HAR models enable accurate classification in various complex application scenarios. Nevertheless, the large storage and computational overhead of the existing fine-grained deep HAR models hinder their widespread deployment on resource-limited platforms. Inspired by the knowledge distillation's reasonable model compression and potential performance improvement capability, we design a multi-level HAR modeling pipeline called Stage-Logits-Memory Distillation (SMLDist) based on the widely-used MobileNet. By paying more attention to the frequency-related features during the distillation process, the SMLDist improves the HAR classification robustness of the students. We also propose an auto-search mechanism in the heterogeneous classifiers to improve classification performance. Extensive simulation results demonstrate that SMLDist outperforms various state-of-the-art HAR frameworks in accuracy and F1 macro score. The practical evaluation of the Jetson Xavier AGX platform shows that the SMLDist model is both energy-efficient and computation-efficient. These experiments validate the reasonable balance between the robustness and efficiency of the proposed model. The comparative experiments of knowledge distillation on six public datasets also demonstrate that the SMLDist outperforms other advanced knowledge distillation methods of students' performance, which verifies the good generalization of the SMLDist on other classification tasks, including but not limited to HAR.