# Temporal Knowledge Distillation for Time-Sensitive Financial Services Applications

arXiv: https://arxiv.org/abs/2312.16799 · HTML: https://mlsanity.com/paper/2312.16799/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Label Refinery: Improving ImageNet Classification through Label Progression](https://mlsanity.com/paper/1805.02641/) (2018)
- [Federated Knowledge Distillation](https://mlsanity.com/paper/2011.02367/) (2020)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training](https://mlsanity.com/paper/1712.02679/) (2017)
- [FireCaffe: near-linear acceleration of deep neural network training on compute clusters](https://mlsanity.com/paper/1511.00175/) (2015)
- [Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More](https://mlsanity.com/paper/2103.10891/) (2021)
- [Deep Q-Network-based Adaptive Alert Threshold Selection Policy for Payment Fraud Systems in Retail Banking](https://mlsanity.com/paper/2010.11062/) (2020)
- [Do Deep Convolutional Nets Really Need to be Deep and Convolutional?](https://mlsanity.com/paper/1603.05691/) (2016)
- [GPU-acceleration for Large-scale Tree Boosting](https://mlsanity.com/paper/1706.08359/) (2017)
- [TKD: Temporal Knowledge Distillation for Active Perception](https://mlsanity.com/paper/1903.01522/) (2019)
- [Essence Knowledge Distillation for Speech Recognition](https://mlsanity.com/paper/1906.10834/) (2019)
- [Computer-Assisted Fraud Detection, From Active Learning to Reward Maximization](https://mlsanity.com/paper/1811.08212/) (2018)

## Built on by (most cited first, 1 of 1)

- [RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2506.00789/) (2025)

## Abstract (arXiv)

Detecting anomalies has become an increasingly critical function in the financial service industry. Anomaly detection is frequently used in key compliance and risk functions such as financial crime detection fraud and cybersecurity. The dynamic nature of the underlying data patterns especially in adversarial environments like fraud detection poses serious challenges to the machine learning models. Keeping up with the rapid changes by retraining the models with the latest data patterns introduces pressures in balancing the historical and current patterns while managing the training data size. Furthermore the model retraining times raise problems in time-sensitive and high-volume deployment systems where the retraining period directly impacts the models ability to respond to ongoing attacks in a timely manner. In this study we propose a temporal knowledge distillation-based label augmentation approach (TKD) which utilizes the learning from older models to rapidly boost the latest model and effectively reduces the model retraining times to achieve improved agility. Experimental results show that the proposed approach provides advantages in retraining times while improving the model performance.