# Knowledge Distillation-Empowered Digital Twin for Anomaly Detection

arXiv: https://arxiv.org/abs/2309.04616 · HTML: https://mlsanity.com/paper/2309.04616/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 8 of its 47 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems](https://mlsanity.com/paper/2012.03448/) (2020)
- [Deep Learning-Based Anomaly Detection in Cyber-Physical Systems: Progress and Opportunities](https://mlsanity.com/paper/2003.13213/) (2020)
- [Packet2Vec: Utilizing Word2Vec for Feature Extraction in Packet Data](https://mlsanity.com/paper/2004.14477/) (2020)
- [EvoCLINICAL: Evolving Cyber-Cyber Digital Twin with Active Transfer Learning for Automated Cancer Registry System](https://mlsanity.com/paper/2309.03246/) (2023)

## Built on by (most cited first, 3 of 3)

- [AI for DevSecOps: A Landscape and Future Opportunities](https://mlsanity.com/paper/2404.04839/) (2024)
- [Digital Twin-based Out-of-Distribution Detection in Autonomous Vessels](https://mlsanity.com/paper/2504.19816/) (2025)
- [DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2610.04494/) (2026)

## Abstract (arXiv)

Cyber-physical systems (CPSs), like train control and management systems (TCMS), are becoming ubiquitous in critical infrastructures. As safety-critical systems, ensuring their dependability during operation is crucial. Digital twins (DTs) have been increasingly studied for this purpose owing to their capability of runtime monitoring and warning, prediction and detection of anomalies, etc. However, constructing a DT for anomaly detection in TCMS necessitates sufficient training data and extracting both chronological and context features with high quality. Hence, in this paper, we propose a novel method named KDDT for TCMS anomaly detection. KDDT harnesses a language model (LM) and a long short-term memory (LSTM) network to extract contexts and chronological features, respectively. To enrich data volume, KDDT benefits from out-of-domain data with knowledge distillation (KD). We evaluated KDDT with two datasets from our industry partner Alstom and obtained the F1 scores of 0.931 and 0.915, respectively, demonstrating the effectiveness of KDDT. We also explored individual contributions of the DT model, LM, and KD to the overall performance of KDDT, via a comprehensive empirical study, and observed average F1 score improvements of 12.4%, 3%, and 6.05%, respectively.