# Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data

arXiv: https://arxiv.org/abs/2407.05315 · HTML: https://mlsanity.com/paper/2407.05315/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 65 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 35)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Persistence Images: A Stable Vector Representation of Persistent Homology](https://mlsanity.com/paper/1507.06217/) (2015)
- [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables](https://mlsanity.com/paper/1604.08880/) (2016)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://mlsanity.com/paper/1907.04829/) (2019)
- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Annealing Knowledge Distillation](https://mlsanity.com/paper/2104.07163/) (2021)
- [Confidence-Aware Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2201.00007/) (2021)
- [Topological Attention for Time Series Forecasting](https://mlsanity.com/paper/2107.09031/) (2021)
- [PI-Net: A Deep Learning Approach to Extract Topological Persistence Images](https://mlsanity.com/paper/1906.01769/) (2019)
- [Generative Adversarial Super-Resolution at the Edge with Knowledge Distillation](https://mlsanity.com/paper/2209.03355/) (2022)
- [Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning](https://mlsanity.com/paper/1907.09554/) (2019)
- [Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast-Choose Three](https://mlsanity.com/paper/2010.06721/) (2020)
- [Transforming Sensor Data to the Image Domain for Deep Learning - an Application to Footstep Detection](https://mlsanity.com/paper/1701.01077/) (2017)
- [Time Series Data Cleaning with Regular and Irregular Time Intervals](https://mlsanity.com/paper/2004.08284/) (2020)
- [Role of Orthogonality Constraints in Improving Properties of Deep Networks for Image Classification](https://mlsanity.com/paper/2009.10762/) (2020)

## Abstract (arXiv)

Deep learning methods have achieved a lot of success in various applications involving converting wearable sensor data to actionable health insights. A common application areas is activity recognition, where deep-learning methods still suffer from limitations such as sensitivity to signal quality, sensor characteristic variations, and variability between subjects. To mitigate these issues, robust features obtained by topological data analysis (TDA) have been suggested as a potential solution. However, there are two significant obstacles to using topological features in deep learning: (1) large computational load to extract topological features using TDA, and (2) different signal representations obtained from deep learning and TDA which makes fusion difficult. In this paper, to enable integration of the strengths of topological methods in deep-learning for time-series data, we propose to use two teacher networks, one trained on the raw time-series data, and another trained on persistence images generated by TDA methods. The distilled student model utilizes only the raw time-series data at test-time. This approach addresses both issues. The use of KD with multiple teachers utilizes complementary information, and results in a compact model with strong supervisory features and an integrated richer representation. To assimilate desirable information from different modalities, we design new constraints, including orthogonality imposed on feature correlation maps for improving feature expressiveness and allowing the student to easily learn from the teacher. Also, we apply an annealing strategy in KD for fast saturation and better accommodation from different features, while the knowledge gap between the teachers and student is reduced. Finally, a robust student model is distilled, which uses only the time-series data as an input, while implicitly preserving topological features.