# Retrospective Loss: Looking Back to Improve Training of Deep Neural Networks

arXiv: https://arxiv.org/abs/2006.13593 · HTML: https://mlsanity.com/paper/2006.13593/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2020.

## Built on by (most cited first, 3 of 3)

- [Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values](https://mlsanity.com/paper/2110.04492/) (2021)
- [SARC: Soft Actor Retrospective Critic](https://mlsanity.com/paper/2306.16503/) (2023)
- [Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization](https://mlsanity.com/paper/2310.12713/) (2023)

## Abstract (arXiv)

Deep neural networks (DNNs) are powerful learning machines that have enabled breakthroughs in several domains. In this work, we introduce a new retrospective loss to improve the training of deep neural network models by utilizing the prior experience available in past model states during training. Minimizing the retrospective loss, along with the task-specific loss, pushes the parameter state at the current training step towards the optimal parameter state while pulling it away from the parameter state at a previous training step. Although a simple idea, we analyze the method as well as to conduct comprehensive sets of experiments across domains - images, speech, text, and graphs - to show that the proposed loss results in improved performance across input domains, tasks, and architectures.