# Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

arXiv: https://arxiv.org/abs/2307.03347 · HTML: https://mlsanity.com/paper/2307.03347/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1912.11976/) (2019)
- [Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data](https://mlsanity.com/paper/2005.10996/) (2020)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data](https://mlsanity.com/paper/2203.08321/) (2022)
- [Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation](https://mlsanity.com/paper/2202.03680/) (2022)
- [On Minimum Discrepancy Estimation for Deep Domain Adaptation](https://mlsanity.com/paper/1901.00282/) (2019)
- [On the benefits of knowledge distillation for adversarial robustness](https://mlsanity.com/paper/2203.07159/) (2022)
- [Revisiting Knowledge Distillation: An Inheritance and Exploration Framework](https://mlsanity.com/paper/2107.00181/) (2021)
- [Knowledge Distillation for BERT Unsupervised Domain Adaptation](https://mlsanity.com/paper/2010.11478/) (2020)
- [Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation](https://mlsanity.com/paper/2005.07839/) (2020)

## Built on by (most cited first, 1 of 1)

- [Finding Foundation Models for Time Series Classification with a PreText Task](https://mlsanity.com/paper/2311.14534/) (2023)

## Abstract (arXiv)

For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource-limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called Universal and joint knowledge distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher's and student's representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks.