# Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition

arXiv: https://arxiv.org/abs/2410.02069 · HTML: https://mlsanity.com/paper/2410.02069/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Vision Foundation Models in Remote Sensing: A Survey](https://mlsanity.com/paper/2408.03464/) (2024)
- [Galaxy Zoo DECaLS: Detailed Visual Morphology Measurements from Volunteers and Deep Learning for 314,000 Galaxies](https://mlsanity.com/paper/2102.08414/) (2021)

## Built on by (most cited first, 4 of 4)

- [FeatSharp: Your Vision Model Features, Sharper](https://mlsanity.com/paper/2502.16025/) (2025)
- [OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triad](https://mlsanity.com/paper/2503.18695/) (2025)
- [Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation](https://mlsanity.com/paper/2509.02615/) (2025)
- [TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces](https://mlsanity.com/paper/2503.07851/) (2025)

## Abstract (arXiv)

In this paper, we present a semi-supervised fine-tuning approach designed to improve the performance of pre-trained foundation models on downstream tasks with limited labeled data. By leveraging content-style decomposition within an information-theoretic framework, our method enhances the latent representations of pre-trained vision foundation models, aligning them more effectively with specific task objectives and addressing the problem of distribution shift. We evaluate our approach on multiple datasets, including MNIST, its augmented variations (with yellow and white stripes), CIFAR-10, SVHN, and GalaxyMNIST. The experiments show improvements over supervised finetuning baseline of pre-trained models, particularly in low-labeled data regimes, across both frozen and trainable backbones for the majority of the tested datasets.