mlsanityData to 14 Oct 2026

Tracking Anything with Decoupled Video Segmentation

arXiv 2309.03903 ↗First posted 7 Sep 2023cs.CVcited by 197

Not on the map. Why?

Abstract (arXiv)

Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. Code is available at: https://hkchengrex.github.io/Tracking-Anything-with-DEVA

Key facts. Cited by 197 papers in the map (data through 14 Oct 2026). Builds on 61 of its 75 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

61 of 75 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Segment Anythingcited by 8,234
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  9. 2016Modeling Context in Referring Expressionscited by 1,063
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2018Panoptic Segmentationcited by 802
  12. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 552
38 more
  1. 2023Segment Everything Everywhere All at Oncecited by 548
  2. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  3. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  4. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  5. 2019Video Instance Segmentationcited by 393
  6. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  7. 2019Tracking without bells and whistlescited by 382
  8. 2023Segment Anything in High Qualitycited by 376
  9. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  10. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  11. 2023Fast Segment Anythingcited by 290
  12. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 234
  13. 2023Track Anything: Segment Anything Meets Videoscited by 234
  14. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  15. 2021Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentationcited by 203
  16. 2023Segment and Track Anythingcited by 193
  17. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  18. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  19. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  20. 2021Mask2Former for Video Instance Segmentationcited by 171
  21. 2018Video Object Segmentation with Language Referring Expressionscited by 162
  22. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  23. 2020Video Panoptic Segmentationcited by 142
  24. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  25. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  26. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  27. 2020f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentationcited by 117
  28. 2020CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinementcited by 113
  29. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  30. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  31. 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105
  32. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 97
  33. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 96
  34. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  35. 2022Towards Grand Unification of Object Trackingcited by 93
  36. 2019Towards High-Resolution Salient Object Detectioncited by 92
  37. 2020Fully Convolutional Networks for Panoptic Segmentationcited by 92
  38. 2023Segment Anything Meets Point Trackingcited by 81

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 80; 2025: 73; 2026: 38.

197 papers.

Sort:

By area:

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  3. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  4. 2024MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffoldscited by 179
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  6. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  7. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  8. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 94
  9. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  10. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 80
  11. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 78
  12. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 75
38 more
  1. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 69
  2. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 62
  3. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  4. 2024FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimallycited by 58
  5. 2024GenXD: Generating Any 3D and 4D Scenescited by 50
  6. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  7. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  8. 2024LVOS: A Benchmark for Large-scale Long-term Video Object Segmentationcited by 44
  9. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  10. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  11. 2024Gaga: Group Any Gaussians via 3D-aware Memory Bankcited by 41
  12. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  13. 2024TrackGo: A Flexible and Efficient Method for Controllable Video Generationcited by 39
  14. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  15. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  16. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 33
  17. 2025Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splattingcited by 31
  18. 2024SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decompositioncited by 30
  19. 2024Moving Object Segmentation: All You Need Is SAM (and Flow)cited by 30
  20. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 29
  21. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29
  22. 2025Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Videocited by 29
  23. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  24. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 26
  25. 2025ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splattingcited by 25
  26. 2024Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RLcited by 23
  27. 2024Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenescited by 22
  28. 2023Audio-Visual Instance Segmentationcited by 21
  29. 2024Segment Any 4D Gaussianscited by 20
  30. 2025COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splittingcited by 19
  31. 2024StyleSplat: 3D Object Style Transfer with Gaussian Splattingcited by 18
  32. 2024SLGaussian: Fast Language Gaussian Splatting in Sparse Viewscited by 18
  33. 2024CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understandingcited by 16
  34. 2024EAGLE: Egocentric AGgregated Language-video Enginecited by 16
  35. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  36. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  37. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  38. 2024Event-guided Low-light Video Semantic Segmentationcited by 14

50 most cited of 197 All 197 (JSON)