mlsanityData to 7 Oct 2026

Motion-inductive Self-supervised Object Discovery in Videos

arXiv 2210.00221 ↗First posted 1 Oct 2022cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the task of unsupervised object discovery in videos. Previous works have shown promising results via processing optical flows to segment objects. However, taking flow as input brings about two drawbacks. First, flow cannot capture sufficient cues when objects remain static or partially occluded. Second, it is challenging to establish temporal coherency from flow-only input, due to the missing texture information. To tackle these limitations, we propose a model for directly processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation, with the opacity channels being treated as the segmentation. Additionally, to enforce object permanence, we apply temporal consistency loss on the inferred masks from randomly-paired frames, which refer to the motions at different paces, and encourage the model to segment the objects even if they may not move at the current time point. Experimentally, we demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets (DAVIS2016, SegTrackv2, and FBMS-59), while being computationally efficient by avoiding the overhead of computing optical flow as input.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 1 Oct 2022.

Builds on

55 of 78 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  6. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  7. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  8. 2020Object-Centric Learning with Slot Attentioncited by 865
  9. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  10. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  11. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  12. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
38 more
  1. 2021Conditional Object-Centric Learning from Videocited by 247
  2. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  3. 2016One-Shot Video Object Segmentationcited by 233
  4. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  5. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  6. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  7. 2018Tracking Emerges by Colorizing Videoscited by 166
  8. 2016Learning Video Object Segmentation from Static Imagescited by 151
  9. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  10. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  11. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  12. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  13. 2021GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinementcited by 109
  14. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  15. 2017Video Object Segmentation Without Temporal Informationcited by 93
  16. 2018VideoMatch: Matching based Video Object Segmentationcited by 90
  17. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  18. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  19. 2017FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videoscited by 77
  20. 2019Unsupervised Deep Trackingcited by 74
  21. 2019Unsupervised Moving Object Detection via Contextual Information Separationcited by 72
  22. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  23. 2017Learning Video Object Segmentation with Visual Memorycited by 66
  24. 2018A Generative Appearance Model for End-to-end Video Object Segmentationcited by 66
  25. 2022Segmenting Moving Objects via an Object-Centric Layered Representationcited by 63
  26. 2022Deformable Sprites for Unsupervised Video Decompositioncited by 61
  27. 2020Generative Neurosymbolic Machinescited by 57
  28. 2018Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagationcited by 56
  29. 2016Learning Motion Patterns in Videoscited by 53
  30. 2020Learning Physical Graph Representations from Visual Scenescited by 50
  31. 2018CNN in MRF: Video Object Segmentation via Inference in A CNN-Based Higher-Order Spatio-Temporal MRFcited by 48
  32. 2018MaskRNN: Instance Level Video Object Segmentationcited by 48
  33. 2021The Emergence of Objectness: Learning Zero-Shot Segmentation from Videoscited by 45
  34. 2022Discovering Objects that Can Movecited by 44
  35. 2020Learning Video Object Segmentation from Unlabeled Videoscited by 42
  36. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  37. 2019Exploiting Spatial Invariance for Scalable Unsupervised Object Trackingcited by 40
  38. 2021Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representationscited by 36

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 5; 2025: 2.

14 papers.

Sort:

  1. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  3. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 77
  4. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  5. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  6. 2023Bootstrapping Objectness from Videos by Relaxed Common Fate and Visual Groupingcited by 15
  7. 2025Learning segmentation from point trajectoriescited by 14
  8. 2023Appearance-Based Refinement for Object-Centric Motion Segmentationcited by 11
  9. 2023Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentationcited by 10
  10. 2023LOCATE: Self-supervised Object Discovery via Flow-guided Graph-cut and Bootstrapped Self-trainingcited by 8
  11. 2024Rethinking Image-to-Video Adaptation: An Object-centric Perspectivecited by 8
  12. 2023Segmenting the motion components of a video: A long-term unsupervised modelcited by 2
2 more
  1. 2023Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slotscited by 0
  2. 2024Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentationcited by 0