# Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks

arXiv: https://arxiv.org/abs/2403.10097 · HTML: https://mlsanity.com/paper/2403.10097/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses](https://mlsanity.com/paper/2003.08983/) (2020)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning](https://mlsanity.com/paper/2102.06605/) (2021)
- [Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning](https://mlsanity.com/paper/1702.08690/) (2017)
- [Improving Deep Regression with Ordinal Entropy](https://mlsanity.com/paper/2301.08915/) (2023)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Analyzing the Performance of Multilayer Neural Networks for Object Recognition](https://mlsanity.com/paper/1407.1610/) (2014)
- [Max-Mahalanobis Linear Discriminant Analysis Networks](https://mlsanity.com/paper/1802.09308/) (2018)
- [DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration](https://mlsanity.com/paper/2308.12058/) (2023)
- [Regularizing CNN Transfer Learning with Randomised Regression](https://mlsanity.com/paper/1908.05997/) (2019)
- [Shaping Deep Feature Space towards Gaussian Mixture for Visual Classification](https://mlsanity.com/paper/2011.09066/) (2020)

## Built on by (most cited first, 1 of 1)

- [Post-pre-training for Modality Alignment in Vision-Language Foundation Models](https://mlsanity.com/paper/2504.12717/) (2025)

## Abstract (arXiv)

While fine-tuning is a de facto standard method for training deep neural networks, it still suffers from overfitting when using small target datasets. Previous methods improve fine-tuning performance by maintaining knowledge of the source datasets or introducing regularization terms such as contrastive loss. However, these methods require auxiliary source information (e.g., source labels or datasets) or heavy additional computations. In this paper, we propose a simple method called adaptive random feature regularization (AdaRand). AdaRand helps the feature extractors of training models to adaptively change the distribution of feature vectors for downstream classification tasks without auxiliary source information and with reasonable computation costs. To this end, AdaRand minimizes the gap between feature vectors and random reference vectors that are sampled from class conditional Gaussian distributions. Furthermore, AdaRand dynamically updates the conditional distribution to follow the currently updated feature extractors and balance the distance between classes in feature spaces. Our experiments show that AdaRand outperforms the other fine-tuning regularization, which requires auxiliary source information and heavy computation costs.