mlsanityData to 7 Oct 2026

Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers

arXiv 2407.08394 ↗First posted 11 Jul 2024cs.CVcited by 24

On the map: Diffusion models → Video generation

Abstract (arXiv)

We introduce Diff-Tracker, a novel approach for the challenging unsupervised visual tracking task leveraging the pre-trained text-to-image diffusion model. Our main idea is to leverage the rich knowledge encapsulated within the pre-trained diffusion model, such as the understanding of image semantics and structural information, to address unsupervised visual tracking. To this end, we design an initial prompt learner to enable the diffusion model to recognize the tracking target by learning a prompt representing the target. Furthermore, to facilitate dynamic adaptation of the prompt to the target's movements, we propose an online prompt updater. Extensive experiments on five benchmark datasets demonstrate the effectiveness of our proposed method, which also achieves state-of-the-art performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 52 of its 70 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

52 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  10. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  11. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  12. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
38 more
  1. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  2. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 445
  3. 2023Emergent Correspondence from Image Diffusioncited by 437
  4. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  5. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  6. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  7. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  8. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  9. 2017Relation Networks for Object Detectioncited by 305
  10. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  11. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  12. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  13. 2016ECO: Efficient Convolution Operators for Trackingcited by 239
  14. 2020Probabilistic Regression for Visual Trackingcited by 231
  15. 2021Oriented R-CNN for Object Detectioncited by 230
  16. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  17. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  18. 2018Distractor-aware Siamese Networks for Visual Object Trackingcited by 187
  19. 2021SwinTrack: A Simple and Strong Baseline for Transformer Trackingcited by 126
  20. 2020Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimationcited by 120
  21. 2022Transforming Model Prediction for Trackingcited by 117
  22. 2022DiffPose: Toward More Reliable 3D Pose Estimationcited by 99
  23. 2020Deformable Siamese Attention Networks for Visual Object Trackingcited by 97
  24. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  25. 2019Unsupervised Deep Trackingcited by 74
  26. 2022Implicit Motion Handling for Video Camouflaged Object Detectioncited by 60
  27. 2022Robust Visual Tracking by Segmentationcited by 38
  28. 2023SLiMe: Segment Like Mecited by 34
  29. 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
  30. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  31. 2020Unsupervised Deep Representation Learning for Real-Time Trackingcited by 29
  32. 2023Distribution-Aligned Diffusion for Human Mesh Recoverycited by 27
  33. 2021Learning to Track Objects from Unlabeled Videoscited by 21
  34. 2023Unsupervised Domain Adaptation via Domain-Adaptive Diffusioncited by 19
  35. 2023Meta Compositional Referring Expression Segmentationcited by 18
  36. 2022Unsupervised Learning of Accurate Siamese Trackingcited by 16
  37. 2022Distilling Inter-Class Distance for Semantic Segmentationcited by 16
  38. 2023MixPro: Data Augmentation with MaskMix and Progressive Attention Labeling for Vision Transformercited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigmcited by 30
  2. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  3. 2024LTRL: Boosting Long-tail Recognition via Reflective Learningcited by 6
  4. 2025Textured 3D Regenerative Morphing with 3D Diffusion Priorcited by 6
  5. 2025An Image-like Diffusion Method for Human-Object Interaction Detectioncited by 5
  6. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentationcited by 4
  7. 2025Informative Sample Selection Model for Skeleton-based Action Recognition with Limited Training Samplescited by 4
  8. 2026DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generationcited by 4
  9. 2025Modality-Guided Dynamic Graph Fusion and Temporal Diffusion for Self-Supervised RGB-T Trackingcited by 2
  10. 2025Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Trackingcited by 2
  11. 2025SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimationcited by 2
  12. 2026Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognitioncited by 2
12 more
  1. 2024FADE: A Dataset for Detecting Falling Objects around Buildings in Videocited by 1
  2. 2026GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editingcited by 1
  3. 2026Boosting Self-Supervised Tracking with Contextual Prompts and Noise Learningcited by 1
  4. 2026Learning to Track Instance from Single Nature Language Descriptioncited by 1
  5. 2026Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Trackingcited by 1
  6. 2026Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Renderingcited by 1
  7. 2026DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Groundingcited by 1
  8. 2025KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generationcited by 0
  9. 2025Image Diffusion Models Exhibit Emergent Temporal Propagation in Videoscited by 0
  10. 2025Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervisioncited by 0
  11. 2026GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecturecited by 0
  12. 2026End-to-End Self-Supervised RGB-T Tracking without Modality Misleadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Emergent Temporal Correspondences from Video Diffusion Transformers
  2. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Models
  3. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners
  4. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  5. 2023Harnessing Diffusion Models for Visual Perception with Meta Prompts
  6. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks
  7. 2025From Image to Video: An Empirical Study of Diffusion Representations
  8. 2025Probing and Leveraging Video Diffusion Transformer Features for Robust Point Tracking