# Distance-Based Regularisation of Deep Networks for Fine-Tuning

arXiv: https://arxiv.org/abs/2002.08253 · HTML: https://mlsanity.com/paper/2002.08253/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 30 of 48)

- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://mlsanity.com/paper/2109.05687/) (2021)
- [Model Stock: All we need is just a few fine-tuned models](https://mlsanity.com/paper/2403.19522/) (2024)
- [Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity](https://mlsanity.com/paper/2305.08252/) (2023)
- [Improved Regularization and Robustness for Fine-tuning in Neural Networks](https://mlsanity.com/paper/2111.04578/) (2021)
- [LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging](https://mlsanity.com/paper/2410.17146/) (2024)
- [Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion](https://mlsanity.com/paper/2302.04451/) (2023)
- [Trainable Projected Gradient Method for Robust Fine-tuning](https://mlsanity.com/paper/2303.10720/) (2023)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees](https://mlsanity.com/paper/2206.02659/) (2022)
- [Fast Trainable Projection for Robust Fine-Tuning](https://mlsanity.com/paper/2310.19182/) (2023)
- [Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approach](https://mlsanity.com/paper/2306.08553/) (2023)
- [Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance](https://mlsanity.com/paper/2404.00860/) (2024)
- [ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections](https://mlsanity.com/paper/2405.20271/) (2024)
- [Directional Gradient Projection for Robust Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2502.15895/) (2025)
- [Learning an Explicit Hyperparameter Prediction Function Conditioned on Tasks](https://mlsanity.com/paper/2107.02378/) (2021)
- [On Measuring Excess Capacity in Neural Networks](https://mlsanity.com/paper/2202.08070/) (2022)
- [Self-Distillation for Further Pre-training of Transformers](https://mlsanity.com/paper/2210.02871/) (2022)
- [Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2411.01713/) (2024)
- [PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocation](https://mlsanity.com/paper/2503.06901/) (2025)
- [Amortised Invariance Learning for Contrastive Self-Supervision](https://mlsanity.com/paper/2302.12712/) (2023)
- [On the Generalization Ability of Unsupervised Pretraining](https://mlsanity.com/paper/2403.06871/) (2024)
- [Meta Mirror Descent: Optimiser Learning for Fast Convergence](https://mlsanity.com/paper/2203.02711/) (2022)
- [PAC-Net: A Model Pruning Approach to Inductive Transfer Learning](https://mlsanity.com/paper/2206.05703/) (2022)
- [ROSE: Robust Selective Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2210.09658/) (2022)
- [DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration](https://mlsanity.com/paper/2308.12058/) (2023)
- [FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering](https://mlsanity.com/paper/2505.21755/) (2025)
- [Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement](https://mlsanity.com/paper/2410.02147/) (2024)
- [Memory-Efficient Fine-Tuning of Transformers via Token Selection](https://mlsanity.com/paper/2501.18824/) (2025)

## Abstract (arXiv)

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.