mlsanityData to 14 Oct 2026

Distance-Based Regularisation of Deep Networks for Fine-Tuning

arXiv 2002.08253 ↗First posted 19 Feb 2020stat.ML cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 11; 2024: 11; 2025: 10; 2026: 3.

48 papers.

Sort:

By area:

  1. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  2. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  3. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  4. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  5. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 56
  6. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  7. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  8. 2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 40
  9. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  10. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
  11. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  12. 2023Fast Trainable Projection for Robust Fine-Tuningcited by 19
36 more
  1. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
  2. 2024Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidancecited by 14
  3. 2024ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflectionscited by 12
  4. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 11
  5. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  6. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  7. 2022Self-Distillation for Further Pre-training of Transformerscited by 9
  8. 2024Rethinking Weight Decay for Robust Fine-Tuning of Foundation Modelscited by 9
  9. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9
  10. 2023Amortised Invariance Learning for Contrastive Self-Supervisioncited by 8
  11. 2024On the Generalization Ability of Unsupervised Pretrainingcited by 8
  12. 2022Meta Mirror Descent: Optimiser Learning for Fast Convergencecited by 7
  13. 2022PAC-Net: A Model Pruning Approach to Inductive Transfer Learningcited by 6
  14. 2022ROSE: Robust Selective Fine-tuning for Pre-trained Language Modelscited by 6
  15. 2023DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibrationcited by 6
  16. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  17. 2024Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglementcited by 4
  18. 2025Memory-Efficient Fine-Tuning of Transformers via Token Selectioncited by 4
  19. 2023Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptabilitycited by 3
  20. 2024AutoFT: Learning an Objective for Robust Fine-Tuningcited by 3
  21. 2024Control Theoretic Approach to Fine-Tuning and Transfer Learningcited by 3
  22. 2022Incremental Few-Shot Learning via Implanting and Compressingcited by 2
  23. 2023Revisiting Fine-Tuning Strategies for Self-supervised Medical Imaging Analysiscited by 2
  24. 2025Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learningcited by 2
  25. 2025VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuningcited by 2
  26. 2024CLIP's Visual Embedding Projector is a Few-shot Cornucopiacited by 1
  27. 2026Model soups need only one ingredientcited by 1
  28. 2023Hierarchical Memory Pool Based Edge Semi-Supervised Continual Learning Methodcited by 0
  29. 2023Examining the Effect of Pre-training on Time Series Classificationcited by 0
  30. 2024Reducing Bias in Pre-trained Models by Tuning while Penalizing Changecited by 0
  31. 2025Norm-Bounded Low-Rank Adaptationcited by 0
  32. 2025LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuningcited by 0
  33. 2025Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphscited by 0
  34. 2025Merge and Bound: Direct Manipulations on Weights for Class Incremental Learningcited by 0
  35. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Modelscited by 0
  36. 2026Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descentcited by 0