# Stereo Vision-based Semantic 3D Object and Ego-motion Tracking for Autonomous Driving

arXiv: https://arxiv.org/abs/1807.02062 · HTML: https://mlsanity.com/paper/1807.02062/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2018.

## Built on by (most cited first, 26 of 26)

- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images](https://mlsanity.com/paper/2403.01517/) (2024)
- [SIDE: Center-based Stereo 3D Detector with Structure-aware Instance Depth Estimation](https://mlsanity.com/paper/2108.09663/) (2021)
- [Stereo CenterNet based 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2103.11071/) (2021)
- [Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration](https://mlsanity.com/paper/2105.14683/) (2021)
- [MOTSLAM: MOT-assisted monocular dynamic SLAM using single-view depth estimation](https://mlsanity.com/paper/2210.02038/) (2022)
- [AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach](https://mlsanity.com/paper/2112.04974/) (2021)
- [Semantic Visual Simultaneous Localization and Mapping: A Survey](https://mlsanity.com/paper/2209.06428/) (2022)
- [3D Video Object Detection with Learnable Object-Centric Global Optimization](https://mlsanity.com/paper/2303.15416/) (2023)
- [Global Correlation Network: End-to-End Joint Multi-Object Detection and Tracking](https://mlsanity.com/paper/2103.12511/) (2021)
- [FocusFlow: Boosting Key-Points Optical Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2308.07104/) (2023)
- [Localization and Mapping using Instance-specific Mesh Models](https://mlsanity.com/paper/2103.04493/) (2021)
- [View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements](https://mlsanity.com/paper/2111.05060/) (2021)
- [Level Set-Based Camera Pose Estimation From Multiple 2D/3D Ellipse-Ellipsoid Correspondences](https://mlsanity.com/paper/2207.07953/) (2022)
- [An End-to-End Framework of Road User Detection, Tracking, and Prediction from Monocular Images](https://mlsanity.com/paper/2308.05026/) (2023)
- [You Only Label Once: 3D Box Adaptation from Point Cloud to Image via Semi-Supervised Learning](https://mlsanity.com/paper/2211.09302/) (2022)
- [UniQuadric: A SLAM Backend for Unknown Rigid Object 3D Tracking and Light-Weight Modeling](https://mlsanity.com/paper/2309.17036/) (2023)
- [Jacobian Computation for Cumulative B-Splines on SE(3) and Application to Continuous-Time Object Tracking](https://mlsanity.com/paper/2201.10602/) (2022)
- [Spotlights: Probing Shapes from Spherical Viewpoints](https://mlsanity.com/paper/2205.12564/) (2022)
- [A Flexible-Frame-Rate Vision-Aided Inertial Object Tracking System for Mobile Devices](https://mlsanity.com/paper/2210.12476/) (2022)
- [TBPos: Dataset for Large-Scale Precision Visual Localization](https://mlsanity.com/paper/2302.09825/) (2023)
- [Leveraging Object Priors for Point Tracking](https://mlsanity.com/paper/2409.05786/) (2024)
- [NF-SLAM: Effective, Normalizing Flow-supported Neural Field representations for object-level visual SLAM in automotive applications](https://mlsanity.com/paper/2503.11199/) (2025)
- [A Stereo Visual SLAM System Using Object-Level Motion Estimation and Geometric Filtering Based on Cross Disparity](https://mlsanity.com/paper/2607.02005/) (2026)

## Abstract (arXiv)

We propose a stereo vision-based approach for tracking the camera ego-motion and 3D semantic objects in dynamic autonomous driving scenarios. Instead of directly regressing the 3D bounding box using end-to-end approaches, we propose to use the easy-to-labeled 2D detection and discrete viewpoint classification together with a light-weight semantic inference method to obtain rough 3D object measurements. Based on the object-aware-aided camera pose tracking which is robust in dynamic environments, in combination with our novel dynamic object bundle adjustment (BA) approach to fuse temporal sparse feature correspondences and the semantic 3D measurement model, we obtain 3D object pose, velocity and anchored dynamic point cloud estimation with instance accuracy and temporal consistency. The performance of our proposed method is demonstrated in diverse scenarios. Both the ego-motion estimation and object localization are compared with the state-of-of-the-art solutions.