# Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data arXiv: https://arxiv.org/abs/2201.00111 ยท HTML: https://mlsanity.com/paper/2201.00111/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation. - Builds on 27 of its 61 references that are in the map. - First posted on arXiv 1 Jan 2022. ## Placed in - Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/) ## Builds on (most cited first, 27 of 27) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016) - [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015) - [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018) - [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019) - [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019) - [Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs](https://mlsanity.com/paper/1706.02633/) (2017) - [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020) - [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015) - [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019) - [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017) - [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019) - [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015) - [RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks](https://mlsanity.com/paper/2002.09545/) (2020) - [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017) - [Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students](https://mlsanity.com/paper/1805.05551/) (2018) - [Knowledge Distillation with Adversarial Samples Supporting Decision Boundary](https://mlsanity.com/paper/1805.05532/) (2018) - [GRATIS: GeneRAting TIme Series with diverse and controllable characteristics](https://mlsanity.com/paper/1903.02787/) (2019) - [Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art](https://mlsanity.com/paper/1806.05226/) (2018) ## Built on by (most cited first, 5 of 5) - [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024) - [Leveraging Angular Distributions for Improved Knowledge Distillation](https://mlsanity.com/paper/2302.14130/) (2023) - [Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data](https://mlsanity.com/paper/2407.05315/) (2024) - [Ground Reaction Force Estimation via Time-aware Knowledge Distillation](https://mlsanity.com/paper/2506.10265/) (2025) - [Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation](https://mlsanity.com/paper/2605.00888/) (2026) ## Abstract (arXiv) Deep neural networks are parametrized by several thousands or millions of parameters, and have shown tremendous success in many classification problems. However, the large number of parameters makes it difficult to integrate these models into edge devices such as smartphones and wearable devices. To address this problem, knowledge distillation (KD) has been widely employed, that uses a pre-trained high capacity network to train a much smaller network, suitable for edge devices. In this paper, for the first time, we study the applicability and challenges of using KD for time-series data for wearable devices. Successful application of KD requires specific choices of data augmentation methods during training. However, it is not yet known if there exists a coherent strategy for choosing an augmentation approach during KD. In this paper, we report the results of a detailed study that compares and contrasts various common choices and some hybrid data augmentation strategies in KD based human activity analysis. Research in this area is often limited as there are not many comprehensive databases available in the public domain from wearable devices. Our study considers databases from small scale publicly available to one derived from a large scale interventional study into human activity and sedentary behavior. We find that the choice of data augmentation techniques during KD have a variable level of impact on end performance, and find that the optimal network choice as well as data augmentation strategies are specific to a dataset at hand. However, we also conclude with a general set of recommendations that can provide a strong baseline performance across databases.