# Combined Image- and World-Space Tracking in Traffic Scenes

arXiv: https://arxiv.org/abs/1809.07357 · HTML: https://mlsanity.com/paper/1809.07357/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 29 of 29)

- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?](https://mlsanity.com/paper/2108.09518/) (2021)
- [4D Panoptic LiDAR Segmentation](https://mlsanity.com/paper/2102.12472/) (2021)
- [QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking](https://mlsanity.com/paper/2210.06984/) (2022)
- [OVTrack: Open-Vocabulary Multiple Object Tracking](https://mlsanity.com/paper/2304.08408/) (2023)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion](https://mlsanity.com/paper/2212.01247/) (2022)
- [PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking?](https://mlsanity.com/paper/2208.01957/) (2022)
- [Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?](https://mlsanity.com/paper/2210.07681/) (2022)
- [A Lightweight and Detector-free 3D Single Object Tracker on Point Clouds](https://mlsanity.com/paper/2203.04232/) (2022)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2311.00987/) (2023)
- [DeconfuseTrack:Dealing with Confusion for Multi-Object Tracking](https://mlsanity.com/paper/2403.02767/) (2024)
- [Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration](https://mlsanity.com/paper/2105.14683/) (2021)
- [DirectTracker: 3D Multi-Object Tracking Using Direct Image Alignment and Photometric Bundle Adjustment](https://mlsanity.com/paper/2209.14965/) (2022)
- [ViewBirdiformer: Learning to recover ground-plane crowd trajectories and ego-motion from a single ego-centric view](https://mlsanity.com/paper/2210.06332/) (2022)
- [An End-to-End Framework of Road User Detection, Tracking, and Prediction from Monocular Images](https://mlsanity.com/paper/2308.05026/) (2023)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [Transformer-based assignment decision network for multiple object tracking](https://mlsanity.com/paper/2208.03571/) (2022)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [ShaSTA-Fuse: Camera-LiDAR Sensor Fusion to Model Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking](https://mlsanity.com/paper/2310.02532/) (2023)
- [Inverse Neural Rendering for Explainable Multi-Object Tracking](https://mlsanity.com/paper/2404.12359/) (2024)
- [One Homography is All You Need: IMM-based Joint Homography and Multiple Object State Estimation](https://mlsanity.com/paper/2409.02562/) (2024)
- [MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking](https://mlsanity.com/paper/2102.03512/) (2021)
- [Automated Construction of Time-Space Diagrams for Traffic Analysis Using Street-View Video Sequence](https://mlsanity.com/paper/2308.06098/) (2023)
- [FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification](https://mlsanity.com/paper/2503.09873/) (2025)
- [Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking](https://mlsanity.com/paper/2609.00924/) (2026)

## Abstract (arXiv)

Tracking in urban street scenes plays a central role in autonomous systems such as self-driving cars. Most of the current vision-based tracking methods perform tracking in the image domain. Other approaches, eg based on LIDAR and radar, track purely in 3D. While some vision-based tracking methods invoke 3D information in parts of their pipeline, and some 3D-based methods utilize image-based information in components of their approach, we propose to use image- and world-space information jointly throughout our method. We present our tracking pipeline as a 3D extension of image-based tracking. From enhancing the detections with 3D measurements to the reported positions of every tracked object, we use world-space 3D information at every stage of processing. We accomplish this by our novel coupled 2D-3D Kalman filter, combined with a conceptually clean and extendable hypothesize-and-select framework. Our approach matches the current state-of-the-art on the official KITTI benchmark, which performs evaluation in the 2D image domain only. Further experiments show significant improvements in 3D localization precision by enabling our coupled 2D-3D tracking.