# VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

arXiv: https://arxiv.org/abs/2510.20994 · HTML: https://mlsanity.com/paper/2510.20994/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 30 of 36)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Optimal Transport Aggregation for Visual Place Recognition](https://mlsanity.com/paper/2311.15937/) (2023)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [Towards Geospatial Foundation Models via Continual Pretraining](https://mlsanity.com/paper/2302.04476/) (2023)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation](https://mlsanity.com/paper/2103.16765/) (2021)
- [Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation](https://mlsanity.com/paper/2003.02824/) (2020)
- [CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition](https://mlsanity.com/paper/2402.19231/) (2024)
- [Self-supervised Domain Adaptation for Computer Vision Tasks](https://mlsanity.com/paper/1907.10915/) (2019)
- [SCENIC: A JAX Library for Computer Vision Research and Beyond](https://mlsanity.com/paper/2110.11403/) (2021)
- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [Unsupervised Learning of Spatiotemporally Coherent Metrics](https://mlsanity.com/paper/1412.6056/) (2014)
- [TIPS: Text-Image Pretraining with Spatial awareness](https://mlsanity.com/paper/2410.16512/) (2024)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks](https://mlsanity.com/paper/2312.02366/) (2023)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)

## Abstract (arXiv)

Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring annotations, leveraging only short multi-view object-centric videos. Our method is referred to as VESSA: Video-based objEct-centric Self-Supervised Adaptation for visual foundation models. VESSA's training technique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficient adaptation techniques - otherwise, the model may quickly forget its pretrained knowledge and reach a degraded state. VESSA benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to varied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available at https://github.com/jesimonbarreto/VESSA.