mlsanityData to 14 Oct 2026

Studying Image Diffusion Features for Zero-Shot Video Object Segmentation

arXiv 2504.05468 ↗First posted 7 Apr 2025cs.CVcited by 4

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

This paper investigates the use of large-scale diffusion models for Zero-Shot Video Object Segmentation (ZS-VOS) without fine-tuning on video data or training on any image segmentation data. While diffusion models have demonstrated strong visual representations across various tasks, their direct application to ZS-VOS remains underexplored. Our goal is to find the optimal feature extraction process for ZS-VOS by identifying the most suitable time step and layer from which to extract features. We further analyze the affinity of these features and observe a strong correlation with point correspondences. Through extensive experiments on DAVIS-17 and MOSE, we find that diffusion models trained on ImageNet outperform those trained on larger, more diverse datasets for ZS-VOS. Additionally, we highlight the importance of point correspondences in achieving high segmentation accuracy, and we yield state-of-the-art results in ZS-VOS. Finally, our approach performs on par with models trained on expensive image segmentation datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 87 of its 105 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

87 of 105 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models22 Jan
  2. 2023SLiMe: Segment Like Me6 Sep
  3. 2023Prompting Diffusion Representations for Cross-Domain Semantic Segmentation5 Jul
  4. 2023LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentation22 Mar
  5. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models21 Mar
  6. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  7. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Models1 Dec

Other references · 80

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2023Segment Anythingcited by 8,234
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  14. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  15. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  16. 2023Segment Everything Everywhere All at Oncecited by 548
  17. 2023Emergent Correspondence from Image Diffusioncited by 446
  18. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  19. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  20. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  21. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  22. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  23. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  24. 2023Personalize Segment Anything Model with One Shotcited by 250
  25. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  26. 2016One-Shot Video Object Segmentationcited by 235
  27. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 234
  28. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 232
  29. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  30. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 203
  31. 2023SegGPT: Segmenting Everything In Contextcited by 194
  32. 2023Putting the Object Back into Video Object Segmentationcited by 188
  33. 2018Video Object Segmentation with Language Referring Expressionscited by 162
  34. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  35. 2016Learning Video Object Segmentation from Static Imagescited by 151
  36. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 148
  37. 2023Matcher: Segment Anything with One Shot Using All-Purpose Feature Matchingcited by 142
  38. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  2. 2025From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Modelscited by 8
  3. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  4. 2025Training-Free Synthetic Data Generation with Dual IP-Adapter Guidancecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  2. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
  3. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation
  4. 2025Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos
  5. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion
  6. 2023Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation
  7. 2024Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning
  8. 2024ZoDi: Zero-Shot Domain Adaptation with Diffusion-Based Image Transfer