mlsanityData to 14 Oct 2026

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

arXiv 2608.07579 ↗First posted 4 Aug 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 19 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

18 of 19 references.

References · 18

  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  5. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  6. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
  7. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 823
  8. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  9. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 552
  10. 2020Deep Learning for Person Re-identification: A Survey and Outlookcited by 549
  11. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 533
  12. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
6 more
  1. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 365
  2. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  3. 2022Slicing Aided Hyper Inference and Fine-tuning for Small Object Detectioncited by 105
  4. 2025Efficiently Reconstructing Dynamic Scenes One D4RT at a Timecited by 54
  5. 2023V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detectioncited by 35
  6. 2023The 7th AI City Challengecited by 20

Built on by

Not cited yet.