mlsanityData to 7 Oct 2026

Track Anything Behind Everything: Zero-Shot Amodal Video Object Segmentation

arXiv 2411.19210 ↗First posted 28 Nov 2024cs.CVcited by 0

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

We present Track Anything Behind Everything (TABE), a novel pipeline for zero-shot amodal video object segmentation. Unlike existing methods that require pretrained class labels, our approach uses a single query mask from the first frame where the object is visible, enabling flexible, zero-shot inference. We pose amodal segmentation as generative outpainting from modal (visible) masks using a pretrained video diffusion model. We do not need to re-train the diffusion model to accommodate additional input channels but instead use a pretrained model that we fine-tune at test-time to allow specialisation towards the tracked object. Our TABE pipeline is specifically designed to handle amodal completion, even in scenarios where objects are completely occluded. Our model and code will all be released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 47 of its 64 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

47 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholes25 Jan

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
34 more
  1. 2024Depth Anything V2cited by 1,385
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  4. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  5. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  6. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
  7. 2019Tracking without bells and whistlescited by 382
  8. 2022Kubric: A scalable dataset generatorcited by 373
  9. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  10. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  11. 2021Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayerscited by 104
  12. 2015Semantic Amodal Segmentationcited by 102
  13. 2018Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentationcited by 62
  14. 2017SeGAN: Segmenting and Generating the Invisiblecited by 60
  15. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
  16. 2024CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibilitycited by 49
  17. 2020Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusioncited by 47
  18. 2023Amodal Ground Truth and Completion in the Wildcited by 43
  19. 2020Amodal Segmentation Based on Visible Region Segmentation and Shape Priorcited by 42
  20. 2022Amodal Panoptic Segmentationcited by 37
  21. 2022AISFormer: Amodal Instance Segmentation with Transformercited by 35
  22. 2020Learning Object Permanence from Videocited by 29
  23. 2021Unsupervised Object Learning via Common Fatecited by 29
  24. 2023Coarse-to-Fine Amodal Segmentation with Shape Priorcited by 29
  25. 2023A Survey of Diffusion Based Image Generation Models: Issues and Their Solutionscited by 23
  26. 2018MVTec D2S: Densely Segmented Supermarket Datasetcited by 20
  27. 2019Robustness of Object Recognition under Extreme Occlusion in Humans and Computational Modelscited by 18
  28. 2022Image Amodal Completion: A Surveycited by 18
  29. 2023Tracking through Containers and Occluders in the Wildcited by 15
  30. 2022Self-supervised Amodal Video Object Segmentationcited by 14
  31. 2023Occluded Person Re-Identification with Deep Learning: A Survey and Perspectivescited by 13
  32. 2023Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representationcited by 9
  33. 2023TAO-Amodal: A Benchmark for Tracking Any Object Amodallycited by 7
  34. 2024Foundation Models for Amodal Video Instance Segmentation in Automated Drivingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Using Diffusion Priors for Video Amodal Segmentation
  2. 2026iSEE: Object Permanence Through Self-Supervision
  3. 2023AnyDoor: Zero-shot Object-level Image Customization
  4. 2026Advancing Complex Video Object Segmentation via Tracking-Enhanced Prompt: The 1st Winner for 5th PVUW MOSE Challenge
  5. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models
  6. 2025Single Point, Full Mask: Velocity-Guided Level Set Evolution for End-to-End Amodal Segmentation
  7. 2024SMITE: Segment Me In TimE
  8. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentation