Data to 7 Oct 2026

Explicit Inductive Bias for Transfer Learning with Convolutional Networks

arXiv 1802.01483 ↗First posted 5 Feb 2018cs.LGcited by 226

Not on the map. Why?

Abstract (arXiv)

In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We show the benefit of having an explicit inductive bias towards the initial model, and we eventually recommend a simple L2 penalty with the pre-trained model being a reference as the baseline of penalty for transfer learning tasks.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 36; 2023: 43; 2024: 49; 2025: 37; 2026: 33.

226 papers.

Sort:

By area:

2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
2021Robust fine-tuning of zero-shot modelscited by 785
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
2023Mitigating the Alignment Tax of RLHFcited by 181
2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
2022Deep Model Reassemblycited by 78
2021On Inductive Biases for Heterogeneous Treatment Effect Estimationcited by 74

38 more

2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
2021Self-Tuning for Data-Efficient Deep Learningcited by 39
2021Adaptive Consistency Regularization for Semi-Supervised Transfer Learningcited by 37
2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
2025Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Datacited by 36
2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
2021Rethinking Supervised Pre-training for Better Downstream Transferringcited by 35
2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
2021Refining Language Models with Compositional Explanationscited by 27
2021TransTailor: Pruning the Pre-trained Model for Improved Transfer Learningcited by 24
2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21
2021How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolationcited by 19
2022Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transportcited by 19
2023Exploring Incompatible Knowledge Transfer in Few-shot Image Generationcited by 19
2024InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integrationcited by 19
2024TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Treescited by 19
2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
2023Fast Trainable Projection for Robust Fine-Tuningcited by 18
2022Harnessing Hard Mixed Samples with Decoupled Regularizercited by 17
2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 17
2022Few-Shot Adaptation of Pre-Trained Networks for Domain Shiftcited by 16
2023SemiReward: A General Reward Model for Semi-supervised Learningcited by 14
2024Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidancecited by 14
2024Comparison of fine-tuning strategies for transfer learning in medical image classificationcited by 14
2024Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalizationcited by 14
2024Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modificationcited by 14
2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
2022Deep transfer learning for image classification: a surveycited by 13

50 most cited of 226 All 226 (JSON)