# VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentation

arXiv: https://arxiv.org/abs/2410.21304 · HTML: https://mlsanity.com/paper/2410.21304/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 61 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications](https://mlsanity.com/paper/2304.05750/) (2023)
- [The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot](https://mlsanity.com/paper/2306.16623/) (2023)
- [Adapting Segment Anything Model for Change Detection in HR Remote Sensing Images](https://mlsanity.com/paper/2309.01429/) (2023)
- [Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications](https://mlsanity.com/paper/2401.06197/) (2024)
- [Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2003.00343/) (2020)
- [The Domain Shift Problem of Medical Image Segmentation and Vendor-Adaptation by Unet-GAN](https://mlsanity.com/paper/1910.13681/) (2019)
- [DeepSeg: Deep Neural Network Framework for Automatic Brain Tumor Segmentation using Magnetic Resonance FLAIR Images](https://mlsanity.com/paper/2004.12333/) (2020)
- [ImageJ2: ImageJ for the next generation of scientific image data](https://mlsanity.com/paper/1701.05940/) (2017)
- [Complementary Domain Adaptation and Generalization for Unsupervised Continual Domain Shift Learning](https://mlsanity.com/paper/2303.15833/) (2023)
- [Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)](https://mlsanity.com/paper/2311.08077/) (2023)
- [Bubble identification from images with machine learning methods](https://mlsanity.com/paper/2202.03107/) (2022)
- [Mitigating Both Covariate and Conditional Shift for Domain Generalization](https://mlsanity.com/paper/2209.08253/) (2022)
- [SAM-PD: How Far Can SAM Take Us in Tracking and Segmenting Anything in Videos by Prompt Denoising](https://mlsanity.com/paper/2403.04194/) (2024)
- [Automatic segmentation of CT images for ventral body composition analysis](https://mlsanity.com/paper/2009.01965/) (2020)
- [Robust and Explainable Fine-Grained Visual Classification with Transfer Learning: A Dual-Carriageway Framework](https://mlsanity.com/paper/2405.05853/) (2024)

## Built on by (most cited first, 1 of 1)

- [MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data](https://mlsanity.com/paper/2411.07463/) (2024)

## Abstract (arXiv)

High-speed video (HSV) segmentation is essential for analyzing dynamic physical processes in scientific and industrial applications, such as boiling heat transfer. Existing models like U-Net struggle with generalization and accurately segmenting complex bubble formations. We present VideoSAM, a specialized adaptation of the Segment Anything Model (SAM), fine-tuned on a diverse HSV dataset for phase detection. Through diverse experiments, VideoSAM demonstrates superior performance across four fluid environments -- Water, FC-72, Nitrogen, and Argon -- significantly outperforming U-Net in complex segmentation tasks. In addition to introducing VideoSAM, we contribute an open-source HSV segmentation dataset designed for phase detection, enabling future research in this domain. Our findings underscore VideoSAM's potential to set new standards in robust and accurate HSV segmentation. The code and dataset used in this study are available online at https://github.com/chikap421/videosam.