# RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr

arXiv: https://arxiv.org/abs/2007.03349 · HTML: https://mlsanity.com/paper/2007.03349/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 27 of 27)

- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning](https://mlsanity.com/paper/2102.06605/) (2021)
- [Fortuitous Forgetting in Connectionist Networks](https://mlsanity.com/paper/2202.00155/) (2022)
- [Adaptive Consistency Regularization for Semi-Supervised Transfer Learning](https://mlsanity.com/paper/2103.02193/) (2021)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [The Impact of Reinitialization on Generalization in Convolutional Neural Networks](https://mlsanity.com/paper/2109.00267/) (2021)
- [When Does Re-initialization Work?](https://mlsanity.com/paper/2206.10011/) (2022)
- [Co-occurrence is not Factual Association in Language Models](https://mlsanity.com/paper/2409.14057/) (2024)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [Privacy Vulnerability of Split Computing to Data-Free Model Inversion Attacks](https://mlsanity.com/paper/2107.06304/) (2021)
- [Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks](https://mlsanity.com/paper/2303.10455/) (2023)
- [ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2205.05282/) (2022)
- [DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration](https://mlsanity.com/paper/2308.12058/) (2023)
- [Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning](https://mlsanity.com/paper/2310.07996/) (2023)
- [Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training](https://mlsanity.com/paper/2401.14948/) (2024)
- [Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization](https://mlsanity.com/paper/2403.19866/) (2024)
- [Towards Inadequately Pre-trained Models in Transfer Learning](https://mlsanity.com/paper/2203.04668/) (2022)
- [SMILE: Self-Distilled MIxup for Efficient Transfer LEarning](https://mlsanity.com/paper/2103.13941/) (2021)
- [BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimization](https://mlsanity.com/paper/2410.02387/) (2024)
- [Video4MRI: An Empirical Study on Brain Magnetic Resonance Image Analytics with CNN-based Video Classification Frameworks](https://mlsanity.com/paper/2302.12688/) (2023)
- ["Forgetting" in Machine Learning and Beyond: A Survey](https://mlsanity.com/paper/2405.20620/) (2024)
- [Recovering Plasticity of Neural Networks via Soft Weight Rescaling](https://mlsanity.com/paper/2507.04683/) (2025)
- [Adversarial Lagrangian Integrated Contrastive Embedding for Limited Size Datasets](https://mlsanity.com/paper/2210.03261/) (2022)
- [CRAFT: Contextual Re-Activation of Filters for face recognition Training](https://mlsanity.com/paper/2312.00072/) (2023)
- [SWiFT: Soft-Mask Weight Fine-tuning for Bias Mitigation](https://mlsanity.com/paper/2508.18826/) (2025)
- [A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning](https://mlsanity.com/paper/2609.33424/) (2026)

## Abstract (arXiv)

Fine-tuning the deep convolution neural network(CNN) using a pre-trained model helps transfer knowledge learned from larger datasets to the target task. While the accuracy could be largely improved even when the training dataset is small, the transfer learning outcome is usually constrained by the pre-trained model with close CNN weights (Liu et al., 2019), as the backpropagation here brings smaller updates to deeper CNN layers. In this work, we propose RIFLE - a simple yet effective strategy that deepens backpropagation in transfer learning settings, through periodically Re-Initializing the Fully-connected LayEr with random scratch during the fine-tuning procedure. RIFLE brings meaningful updates to the weights of deep CNN layers and improves low-level feature learning, while the effects of randomization can be easily converged throughout the overall learning procedure. The experiments show that the use of RIFLE significantly improves deep transfer learning accuracy on a wide range of datasets, out-performing known tricks for the similar purpose, such as Dropout, DropConnect, StochasticDepth, Disturb Label and Cyclic Learning Rate, under the same settings with 0.5% -2% higher testing accuracy. Empirical cases and ablation studies further indicate RIFLE brings meaningful updates to deep CNN layers with accuracy improved.