# Generative Uncertainty as a Self-supervised Signal for Semantic Similarity Learning

arXiv: https://arxiv.org/abs/2609.35341 · HTML: https://mlsanity.com/paper/2609.35341/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models](https://mlsanity.com/paper/2308.06160/) (2023)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [Denoising diffusion models for out-of-distribution detection](https://mlsanity.com/paper/2211.07740/) (2022)
- [Synthetic Humans for Action Recognition from Unseen Viewpoints](https://mlsanity.com/paper/1912.04070/) (2019)
- [Generative Uncertainty in Diffusion Models](https://mlsanity.com/paper/2502.20946/) (2025)
- [Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models](https://mlsanity.com/paper/2406.18580/) (2024)
- [Diffusion Model Guided Sampling with Pixel-Wise Aleatoric Uncertainty Estimation](https://mlsanity.com/paper/2412.00205/) (2024)
- [UVEB: A Large-scale Benchmark and Baseline Towards Real-World Underwater Video Enhancement](https://mlsanity.com/paper/2404.14542/) (2024)
- [Video Similarity and Alignment Learning on Partial Video Copy Detection](https://mlsanity.com/paper/2108.01817/) (2021)
- [Self-Supervised Video Similarity Learning](https://mlsanity.com/paper/2304.03378/) (2023)
- [Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation](https://mlsanity.com/paper/2412.03178/) (2024)
- [MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding](https://mlsanity.com/paper/2512.06581/) (2025)
- [VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression](https://mlsanity.com/paper/2303.08906/) (2023)
- [ConViS-Bench: Estimating Video Similarity Through Semantic Concepts](https://mlsanity.com/paper/2509.19245/) (2025)
- [VRAG: Region Attention Graphs for Content-Based Video Retrieval](https://mlsanity.com/paper/2205.09068/) (2022)

## Abstract (arXiv)

Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.