# Self-Referenced Deep Learning

arXiv: https://arxiv.org/abs/1811.07598 · HTML: https://mlsanity.com/paper/1811.07598/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2018.

## Built on by (most cited first, 7 of 7)

- [Knowledge-guided Causal Intervention for Weakly-supervised Object Localization](https://mlsanity.com/paper/2301.01060/) (2023)
- [Matryoshka Model Learning for Improved Elastic Student Models](https://mlsanity.com/paper/2505.23337/) (2025)
- [Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation](https://mlsanity.com/paper/2108.11798/) (2021)
- [Domain-Agnostic Clustering with Self-Distillation](https://mlsanity.com/paper/2111.12170/) (2021)
- [Dynamic Rectification Knowledge Distillation](https://mlsanity.com/paper/2201.11319/) (2022)
- [Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation](https://mlsanity.com/paper/2605.00888/) (2026)
- [Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation](https://mlsanity.com/paper/2606.25927/) (2026)

## Abstract (arXiv)

Knowledge distillation is an effective approach to transferring knowledge from a teacher neural network to a student target network for satisfying the low-memory and fast running requirements in practice use. Whilst being able to create stronger target networks compared to the vanilla non-teacher based learning strategy, this scheme needs to train additionally a large teacher model with expensive computational cost. In this work, we present a Self-Referenced Deep Learning (SRDL) strategy. Unlike both vanilla optimisation and existing knowledge distillation, SRDL distils the knowledge discovered by the in-training target model back to itself to regularise the subsequent learning procedure therefore eliminating the need for training a large teacher model. SRDL improves the model generalisation performance compared to vanilla learning and conventional knowledge distillation approaches with negligible extra computational cost. Extensive evaluations show that a variety of deep networks benefit from SRDL resulting in enhanced deployment performance on both coarse-grained object categorisation tasks (CIFAR10, CIFAR100, Tiny ImageNet, and ImageNet) and fine-grained person instance identification tasks (Market-1501).