# DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking

arXiv: https://arxiv.org/abs/2303.16628 · HTML: https://mlsanity.com/paper/2303.16628/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 61 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [Unifying Voxel-based Representation with Transformer for 3D Object Detection](https://mlsanity.com/paper/2206.00630/) (2022)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)

## Built on by (most cited first, 11 of 11)

- [Sparse4D v3: Advancing End-to-End 3D Detection and Tracking](https://mlsanity.com/paper/2311.11722/) (2023)
- [SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving](https://mlsanity.com/paper/2404.06892/) (2024)
- [BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detection](https://mlsanity.com/paper/2312.01696/) (2023)
- [DualAD: Disentangling the Dynamic and Static World for End-to-End Driving](https://mlsanity.com/paper/2406.06264/) (2024)
- [Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Tracking](https://mlsanity.com/paper/2510.10287/) (2025)
- [TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation](https://mlsanity.com/paper/2404.11803/) (2024)
- [RockTrack: A 3D Robust Multi-Camera-Ken Multi-Object Tracking Framework](https://mlsanity.com/paper/2409.11749/) (2024)
- [SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking](https://mlsanity.com/paper/2411.06780/) (2024)
- [Towards Latency-Aware 3D Streaming Perception for Autonomous Driving](https://mlsanity.com/paper/2504.19115/) (2025)
- [Leveraging Previous-Traversal Point Cloud Map Priors for Camera-Based 3D Object Detection and Tracking](https://mlsanity.com/paper/2604.25405/) (2026)
- [Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection](https://mlsanity.com/paper/2409.05425/) (2024)

## Abstract (arXiv)

Recent multi-camera 3D object detectors usually leverage temporal information to construct multi-view stereo that alleviates the ill-posed depth estimation. However, they typically assume all the objects are static and directly aggregate features across frames. This work begins with a theoretical and empirical analysis to reveal that ignoring the motion of moving objects can result in serious localization bias. Therefore, we propose to model Dynamic Objects in RecurrenT (DORT) to tackle this problem. In contrast to previous global Bird-Eye-View (BEV) methods, DORT extracts object-wise local volumes for motion estimation that also alleviates the heavy computational burden. By iteratively refining the estimated object motion and location, the preceding features can be precisely aggregated to the current frame to mitigate the aforementioned adverse effects. The simple framework has two significant appealing properties. It is flexible and practical that can be plugged into most camera-based 3D object detectors. As there are predictions of object motion in the loop, it can easily track objects across frames according to their nearest center distances. Without bells and whistles, DORT outperforms all the previous methods on the nuScenes detection and tracking benchmarks with 62.5% NDS and 57.6% AMOTA, respectively. The source code will be released.