mlsanityData to 7 Oct 2026

CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos

arXiv 2410.11831 ↗First posted 15 Oct 2024cs.CVcited by 122

Not on the map. Why?

Abstract (arXiv)

Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

28 of 41 references.

References · 28

  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  5. 2023CoTracker: It is Better to Track Togethercited by 516
  6. 2022Kubric: A scalable dataset generatorcited by 373
  7. 2020DISK: Learning local features with policy gradientcited by 310
  8. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  9. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  10. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  11. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  12. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
16 more
  1. 2023Visual Geometry Grounded Deep Structure From Motioncited by 225
  2. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  3. 2017UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Losscited by 202
  4. 2018Tracking Emerges by Colorizing Videoscited by 166
  5. 2023DynamicStereo: Consistent Dynamic Depth from Stereo Videoscited by 147
  6. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  7. 2020Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimationcited by 120
  8. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 115
  9. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  10. 2017Occlusion Aware Unsupervised Learning of Optical Flowcited by 101
  11. 2024Local All-Pair Correspondence for Point Trackingcited by 77
  12. 2019DDFlow: Learning Optical Flow with Unlabeled Data Distillationcited by 63
  13. 2023Dense Optical Tracking: Connecting the Dotscited by 58
  14. 2014Unsupervised Learning of Spatiotemporally Coherent Metricscited by 45
  15. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 43
  16. 2024Refining Pre-Trained Motion Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 97.

122 papers.

Sort:

By area:

  1. 2025Puppeteer: Rig and Animate Your 3D Modelscited by 41
  2. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  3. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  4. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  5. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 21
  6. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  7. 2025TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videoscited by 13
  8. 2025EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flowcited by 10
  9. 2026Point Tracking Improves World Action Modelscited by 7
  10. 2026PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motioncited by 7
  11. 2026PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulationcited by 7
  12. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
38 more
  1. 2025ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flowcited by 5
  2. 2025MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learningcited by 5
  3. 2025MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Modelscited by 5
  4. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  5. 2026MegaFlow: Zero-Shot Large Displacement Optical Flowcited by 5
  6. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  7. 2025Self-Supervised Learning of Motion Concepts by Optimizing Counterfactualscited by 4
  8. 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
  9. 2025Prepare Before You Act: Learning From Humans to Rearrange Initial Statescited by 3
  10. 2025Track-On2: Enhancing Online Point Tracking with Memorycited by 3
  11. 2025Probing and Leveraging Video Diffusion Transformer Features for Robust Point Trackingcited by 3
  12. 2026TrajVG: 3D Trajectory-Coupled Visual Geometry Learningcited by 3
  13. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  14. 2026ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactionscited by 3
  15. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instructioncited by 3
  16. 2025VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigationcited by 2
  17. 2025Is This Tracker On? A Benchmark Protocol for Dynamic Trackingcited by 2
  18. 2025PointSt3R: Point Tracking through 3D Grounded Correspondencecited by 2
  19. 2026SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Trackingcited by 2
  20. 2026SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuningcited by 2
  21. 2026On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learningcited by 2
  22. 2026Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wildcited by 2
  23. 2026MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generationcited by 2
  24. 2026ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translationcited by 2
  25. 2026Learning Physics-Guided Residual Dynamics for Deformable Object Simulationcited by 2
  26. 2026DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Modelscited by 2
  27. 2026TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Imagecited by 2
  28. 2025SAVeD: Learning to Denoise Low-SNR Video for Improved Downstream Performancecited by 1
  29. 2025Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generationcited by 1
  30. 2026AeroDGS: Physically Consistent Dynamic Gaussian Splatting for Single-Sequence Aerial 4D Reconstructioncited by 1
  31. 2026Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusioncited by 1
  32. 2026DancingBox: A Lightweight MoCap System for Character Animation from Physical Proxiescited by 1
  33. 2026SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scalecited by 1
  34. 2026SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamicscited by 1
  35. 2026Flow as Flow: Modeling Robot Velocity Fields as Probability Velocity Fields for Flow-Based Object Manipulationcited by 1
  36. 2026Argus: Metric Panoramic 3D Reconstruction for Indoor Scenescited by 1
  37. 2026UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Imagecited by 1
  38. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1

50 most cited of 122 All 122 (JSON)