mlsanityData to 7 Oct 2026

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

arXiv 2201.08379 ↗First posted 20 Jan 2022cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence. Yet, approaches that dominate each space differ. We take a step towards bridging this gap by extending the recent contrastive random walk formulation to much denser, pixel-level space-time graphs. The main contribution is introducing hierarchy into the search problem by computing the transition matrix between two frames in a coarse-to-fine manner, forming a multiscale contrastive random walk when extended in time. This establishes a unified technique for self-supervised learning of optical flow, keypoint tracking, and video object segmentation. Experiments demonstrate that, for each of these tasks, the unified model achieves performance competitive with strong self-supervised approaches specific to that task. Project webpage: https://jasonbian97.github.io/flowwalk

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 42 of its 85 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

42 of 85 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  6. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  8. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  9. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  10. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  11. 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
  12. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
30 more
  1. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  2. 2017UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Losscited by 202
  3. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  4. 2018Tracking Emerges by Colorizing Videoscited by 166
  5. 2014Simultaneous Detection and Segmentationcited by 158
  6. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  7. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  8. 2020What Matters in Unsupervised Optical Flowcited by 124
  9. 2020Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimationcited by 120
  10. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  11. 2021AutoFlow: Learning a Better Training Set for Optical Flowcited by 102
  12. 2016Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothnesscited by 101
  13. 2017Occlusion Aware Unsupervised Learning of Optical Flowcited by 101
  14. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  15. 2015EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flowcited by 80
  16. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  17. 2019Unsupervised Deep Trackingcited by 74
  18. 2021SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warpingcited by 74
  19. 2018Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimationcited by 69
  20. 2019DDFlow: Learning Optical Flow with Unlabeled Data Distillationcited by 63
  21. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  22. 2020Watching the World Go By: Representation Learning from Unlabeled Videoscited by 59
  23. 2020UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learningcited by 49
  24. 2021Warp Consistency for Unsupervised Learning of Dense Correspondencescited by 42
  25. 2021Self-Point-Flow: Self-Supervised Scene Flow Estimation from Point Clouds with Optimal Transport and Random Walkcited by 39
  26. 2019Unsupervised Deep Epipolar Flow for Stationary or Dynamic Scenescited by 29
  27. 2017MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimationcited by 21
  28. 2017Learning by Association - A versatile semi-supervised training method for neural networkscited by 19
  29. 2021LIFE: Lighting Invariant Flow Estimationcited by 6
  30. 2021Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 15; 2024: 9; 2025: 7; 2026: 1.

36 papers.

Sort:

By area:

  1. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  2. 2023Tracking Everything Everywhere All at Oncecited by 190
  3. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  4. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  5. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  6. 2023A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyondcited by 55
  7. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  8. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  9. 2022textttGradICON: Approximate Diffeomorphisms via Gradient Inverse Consistencycited by 40
  10. 2024Local Feature Matching Using Deep Learning: A Surveycited by 26
  11. 2023Object Discovery from Motion-Guided Tokenscited by 25
  12. 2023Boosting Video Object Segmentation via Space-time Correspondence Learningcited by 21
24 more
  1. 2022Self-supervised video pretraining yields robust and more human-aligned visual representationscited by 20
  2. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  3. 2022Sound Localization by Self-Supervised Time Delay Estimationcited by 18
  4. 2023Object-centric Learning with Cyclic Walks between Parts and Wholecited by 16
  5. 2023Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentationcited by 16
  6. 2024Track Everything Everywhere Fast and Robustlycited by 13
  7. 2024Refining Pre-Trained Motion Modelscited by 12
  8. 2023Self-Supervised Motion Magnification by Backpropagating Through Optical Flowcited by 10
  9. 2023Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentationcited by 10
  10. 2024Self-Supervised Any-Point Tracking by Contrastive Random Walkscited by 10
  11. 2023Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Timecited by 5
  12. 2024Made to Order: Discovering monotonic temporal changes via self-supervised video orderingcited by 5
  13. 2025Self-Supervised Learning of Motion Concepts by Optimizing Counterfactualscited by 4
  14. 2025Super Encoding Network: Recursive Association of Multi-Modal Encoders for Video Understandingcited by 4
  15. 2025Taming generative video models for zero-shot optical flow extractioncited by 4
  16. 2023DRIMET: Deep Registration for 3D Incompressible Motion Estimation in Tagged-MRI with Application to the Tonguecited by 2
  17. 2023MomentaMorph: Unsupervised Spatial-Temporal Registration with Momenta, Shooting, and Correctioncited by 2
  18. 2025Self-Supervised Spatial Correspondence Across Modalitiescited by 2
  19. 2024Decomposition Betters Tracking Everything Everywherecited by 1
  20. 2025GMR-Conv: An Efficient Rotation and Reflection Equivariant Convolution Kernel Using Gaussian Mixture Ringscited by 1
  21. 2026Exploring High-Order Self-Similarity for Video Understandingcited by 1
  22. 2023S3Track: Self-supervised Tracking with Soft Assignment Flowcited by 0
  23. 2025Leveraging Cycle-Consistent Anchor Points for Self-Supervised RGB-D Registrationcited by 0
  24. 2025Tracking by Predicting 3-D Gaussians Over Timecited by 0