mlsanityData to 7 Oct 2026

Virtual Worlds as Proxy for Multi-Object Tracking Analysis

arXiv 1605.06457 ↗First posted 20 May 2016cs.CV cs.LG cs.NEcited by 340

Not on the map. Why?

Abstract (arXiv)

Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called Virtual KITTI (see http://www.xrce.xerox.com/Research-Development/Computer-Vision/Proxy-Virtual-Worlds), automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.

Key facts. Cited by 340 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 72; 2022: 58; 2023: 49; 2024: 63; 2025: 52; 2026: 45.

340 papers.

Sort:

By area:

  1. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  2. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  3. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  4. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  5. 2022Kubric: A scalable dataset generatorcited by 373
  6. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  7. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  8. 2021DSEC: A Stereo Event Camera Dataset for Driving Scenarioscited by 297
  9. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  10. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  11. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  12. 2023Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereocited by 170
38 more
  1. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  2. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  3. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  4. 2021Learning to Track with Object Permanencecited by 117
  5. 2025VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequencescited by 107
  6. 2021AutoFlow: Learning a Better Training Set for Optical Flowcited by 102
  7. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  8. 2021Monocular Quasi-Dense 3D Object Trackingcited by 80
  9. 2021RigNet: Repetitive Image Guided Network for Depth Completioncited by 80
  10. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  11. 2021Transfer Learning from Synthetic to Real LiDAR Point Cloud for Semantic Segmentationcited by 76
  12. 20213D Semantic Scene Completion: a Surveycited by 74
  13. 2023Semantic Segmentation using Vision Transformers: A surveycited by 73
  14. 2021STEP: Segmenting and Tracking Every Pixelcited by 72
  15. 2021A Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detectioncited by 68
  16. 2021MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?cited by 68
  17. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  18. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  19. 2021Unsupervised Depth Completion with Calibrated Backprojection Layerscited by 61
  20. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  21. 2021Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Typescited by 54
  22. 2021SSPC-Net: Semi-supervised Semantic 3D Point Cloud Segmentation Networkcited by 54
  23. 2021Transferability Estimation using Bhattacharyya Class Separabilitycited by 53
  24. 2023OVTrack: Open-Vocabulary Multiple Object Trackingcited by 53
  25. 2022STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Datasetcited by 52
  26. 2022Hierarchical Normalization for Robust Monocular Depth Estimationcited by 51
  27. 2022SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Cloudscited by 49
  28. 2022Pin the Memory: Learning to Generalize Semantic Segmentationcited by 48
  29. 2023Synthetic Datasets for Autonomous Driving: A Surveycited by 48
  30. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 46
  31. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  32. 2025Dens3R: A Foundation Model for 3D Geometry Predictioncited by 44
  33. 2025RoMa v2: Harder Better Faster Denser Feature Matchingcited by 44
  34. 2021Toward Practical Monocular Indoor Depth Estimationcited by 42
  35. 2024Tri-Perspective View Decomposition for Geometry-Aware Depth Completioncited by 42
  36. 2024SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentationcited by 41
  37. 2021Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentationcited by 40
  38. 2023All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenescited by 40

50 most cited of 340 All 340 (JSON)