# Targeted Gradient Descent: A Novel Method for Convolutional Neural Networks Fine-tuning and Online-learning

arXiv: https://arxiv.org/abs/2109.14729 · HTML: https://mlsanity.com/paper/2109.14729/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [Unsupervised domain adaptation in brain lesion segmentation with adversarial networks](https://mlsanity.com/paper/1612.08894/) (2016)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation](https://mlsanity.com/paper/1702.07841/) (2017)
- [Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression](https://mlsanity.com/paper/1812.04368/) (2018)
- [Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data](https://mlsanity.com/paper/1810.12448/) (2018)
- [Towards continual learning in medical imaging](https://mlsanity.com/paper/1811.02496/) (2018)
- [Semantic-Aware Generative Adversarial Nets for Unsupervised Domain Adaptation in Chest X-ray Segmentation](https://mlsanity.com/paper/1806.00600/) (2018)
- [Uncertainty Estimation in Medical Image Denoising with Bayesian Deep Image Prior](https://mlsanity.com/paper/2008.08837/) (2020)
- [Distributed Weight Consolidation: A Brain Segmentation Case Study](https://mlsanity.com/paper/1805.10863/) (2018)
- [Fine tuning U-Net for ultrasound image segmentation: which layers?](https://mlsanity.com/paper/2002.08438/) (2020)

## Built on by (most cited first, 2 of 2)

- [DCSAU-Net: A Deeper and More Compact Split-Attention U-Net for Medical Image Segmentation](https://mlsanity.com/paper/2202.00972/) (2022)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)

## Abstract (arXiv)

A convolutional neural network (ConvNet) is usually trained and then tested using images drawn from the same distribution. To generalize a ConvNet to various tasks often requires a complete training dataset that consists of images drawn from different tasks. In most scenarios, it is nearly impossible to collect every possible representative dataset as a priori. The new data may only become available after the ConvNet is deployed in clinical practice. ConvNet, however, may generate artifacts on out-of-distribution testing samples. In this study, we present Targeted Gradient Descent (TGD), a novel fine-tuning method that can extend a pre-trained network to a new task without revisiting data from the previous task while preserving the knowledge acquired from previous training. To a further extent, the proposed method also enables online learning of patient-specific data. The method is built on the idea of reusing a pre-trained ConvNet's redundant kernels to learn new knowledge. We compare the performance of TGD to several commonly used training approaches on the task of Positron emission tomography (PET) image denoising. Results from clinical images show that TGD generated results on par with training-from-scratch while significantly reducing data preparation and network training time. More importantly, it enables online learning on the testing study to enhance the network's generalization capability in real-world applications.