# Learning with Retrospection

arXiv: https://arxiv.org/abs/2012.13098 · HTML: https://mlsanity.com/paper/2012.13098/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [Calibrating a Deep Neural Network with Its Predecessors](https://mlsanity.com/paper/2302.06245/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Reducing Flipping Errors in Deep Neural Networks](https://mlsanity.com/paper/2203.08390/) (2022)
- [Online Continual Learning via Multiple Deep Metric Learning and Uncertainty-guided Episodic Memory Replay -- 3rd Place Solution for ICCV 2021 Workshop SSLAD Track 3A Continual Object Classification](https://mlsanity.com/paper/2111.02757/) (2021)
- [Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding](https://mlsanity.com/paper/2206.02963/) (2022)
- [Improving Model Training via Self-learned Label Representations](https://mlsanity.com/paper/2209.04528/) (2022)

## Abstract (arXiv)

Deep neural networks have been successfully deployed in various domains of artificial intelligence, including computer vision and natural language processing. We observe that the current standard procedure for training DNNs discards all the learned information in the past epochs except the current learned weights. An interesting question is: is this discarded information indeed useless? We argue that the discarded information can benefit the subsequent training. In this paper, we propose learning with retrospection (LWR) which makes use of the learned information in the past epochs to guide the subsequent training. LWR is a simple yet effective training framework to improve accuracies, calibration, and robustness of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead. Extensive experiments on several benchmark datasets demonstrate the superiority of LWR for training DNNs.