# From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning

arXiv: https://arxiv.org/abs/2603.26597 · HTML: https://mlsanity.com/paper/2603.26597/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 122 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [Spatiotemporal Predictive Pre-training for Robotic Motor Control](https://mlsanity.com/paper/2403.05304/) (2024)
- [Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection](https://mlsanity.com/paper/2405.09782/) (2024)
- [Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening](https://mlsanity.com/paper/2509.08502/) (2025)
- [Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning](https://mlsanity.com/paper/2105.05838/) (2021)
- [Exploring Structural Degradation in Dense Representations for Self-supervised Learning](https://mlsanity.com/paper/2510.17299/) (2025)

## Built on by (most cited first, 2 of 2)

- [Training-Free Composed Video Retrieval via Visual Representation-Guided Video-LLM Reasoning](https://mlsanity.com/paper/2606.02321/) (2026)
- [Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?](https://mlsanity.com/paper/2606.03837/) (2026)

## Abstract (arXiv)

Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.