# Lifting Multi-View Detection and Tracking to the Bird's Eye View

arXiv: https://arxiv.org/abs/2403.12573 · HTML: https://mlsanity.com/paper/2403.12573/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Detect to Track and Track to Detect](https://mlsanity.com/paper/1710.03958/) (2017)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification](https://mlsanity.com/paper/1903.09254/) (2019)
- [Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification](https://mlsanity.com/paper/1809.04427/) (2018)
- [Multiview Detection with Feature Perspective Transformation](https://mlsanity.com/paper/2007.07247/) (2020)
- [Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)](https://mlsanity.com/paper/2108.05888/) (2021)
- [Simple Cues Lead to a Strong Multi-Object Tracker](https://mlsanity.com/paper/2206.04656/) (2022)
- [Deep Occlusion Reasoning for Multi-Camera Multi-Target Detection](https://mlsanity.com/paper/1704.05775/) (2017)
- [A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Research](https://mlsanity.com/paper/2204.06527/) (2022)
- [Deep Multi-camera People Detection](https://mlsanity.com/paper/1702.04593/) (2017)
- [Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception](https://mlsanity.com/paper/2403.07746/) (2024)
- [LMGP: Lifted Multicut Meets Geometry Projections for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2111.11892/) (2021)
- [EarlyBird: Early-Fusion for Multi-View Tracking in the Bird's Eye View](https://mlsanity.com/paper/2310.13350/) (2023)
- [ReST: A Reconfigurable Spatial-Temporal Graph Model for Multi-Camera Multi-Object Tracking](https://mlsanity.com/paper/2308.13229/) (2023)

## Built on by (most cited first, 25 of 25)

- [SpaRC: Sparse Radar-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2411.19860/) (2024)
- [Enhanced Multi-View Pedestrian Detection Using Probabilistic Occupancy Volume](https://mlsanity.com/paper/2503.10982/) (2025)
- [MVTrajecter: Multi-View Pedestrian Tracking with Trajectory Motion Cost and Trajectory Appearance Cost](https://mlsanity.com/paper/2509.01157/) (2025)
- [LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs](https://mlsanity.com/paper/2506.11148/) (2025)
- [Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders](https://mlsanity.com/paper/2410.04817/) (2024)
- [Attention-Aware Multi-View Pedestrian Tracking](https://mlsanity.com/paper/2504.03047/) (2025)
- [One Graph to Track Them All: Dynamic GNNs for Single- and Multi-View Tracking](https://mlsanity.com/paper/2507.08494/) (2025)
- [Spatial-Temporal Multi-Cuts for Online Multiple-Camera Vehicle Tracking](https://mlsanity.com/paper/2410.02638/) (2024)
- [Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary](https://mlsanity.com/paper/2412.16511/) (2024)
- [SpaRC-AD: A Baseline for Radar-Camera Fusion in End-to-End Autonomous Driving](https://mlsanity.com/paper/2508.10567/) (2025)
- [ATLASFusion: Aggregation Tracking with Location-Aware Sparse Fusion for Robust Spatio-Temporal Multi-View Pedestrian Tracking](https://mlsanity.com/paper/2509.08421/) (2025)
- [Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey](https://mlsanity.com/paper/2510.09731/) (2025)
- [Advanced Unsupervised Learning: A Comprehensive Overview of Multi-View Clustering Techniques](https://mlsanity.com/paper/2512.05169/) (2025)
- [C-DOG: Multi-View Multi-instance Feature Association Using Connected δ-Overlap Graphs](https://mlsanity.com/paper/2507.14095/) (2025)
- [Stable Multi-Drone GNSS Tracking System for Marine Robots](https://mlsanity.com/paper/2511.18694/) (2025)
- [ModTrack: Sensor-Agnostic Multi-View Tracking via Identity-Informed PHD Filtering with Covariance Propagation](https://mlsanity.com/paper/2603.15812/) (2026)
- [SynMVCrowd: A Large Synthetic Benchmark for Multi-view Crowd Counting and Localization](https://mlsanity.com/paper/2603.23956/) (2026)
- [Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes](https://mlsanity.com/paper/2604.19318/) (2026)
- [Fully Distributed Multi-View 3D Tracking in Real-Time](https://mlsanity.com/paper/2606.13127/) (2026)
- [ARGUSTRACK: A Multi-View Annotation System for Multi-Object Tracking](https://mlsanity.com/paper/2606.20687/) (2026)
- [MVDGC: Joint 3D and 2D Multi-view Pedestrian Detection via Dual Geometric Constraints](https://mlsanity.com/paper/2607.00273/) (2026)
- [Beyond Appearance: A Multi-cue Framework and Large-scale Benchmark for Pedestrian Association and Tracking on Mobile Aerial-Ground Platforms](https://mlsanity.com/paper/2607.23803/) (2026)
- [GRACE: Geometry- and Ray-Aware Camera-Efficient Multi-View Pedestrian Tracking](https://mlsanity.com/paper/2609.16872/) (2026)
- [Online Multi-Camera 3D Tracking via ID Prediction over Recurrent Sparse Queries](https://mlsanity.com/paper/2609.18363/) (2026)
- [CAT-Free: Multi-View Pedestrian Localization without Calibration, Annotations, or Target-Scene Training via Adaptive Geometric Filtering](https://mlsanity.com/paper/2609.34302/) (2026)

## Abstract (arXiv)

Taking advantage of multi-view aggregation presents a promising solution to tackle challenges such as occlusion and missed detection in multi-object tracking and detection. Recent advancements in multi-view detection and 3D object recognition have significantly improved performance by strategically projecting all views onto the ground plane and conducting detection analysis from a Bird's Eye View. In this paper, we compare modern lifting methods, both parameter-free and parameterized, to multi-view aggregation. Additionally, we present an architecture that aggregates the features of multiple times steps to learn robust detection and combines appearance- and motion-based cues for tracking. Most current tracking approaches either focus on pedestrians or vehicles. In our work, we combine both branches and add new challenges to multi-view detection with cross-scene setups. Our method generalizes to three public datasets across two domains: (1) pedestrian: Wildtrack and MultiviewX, and (2) roadside perception: Synthehicle, achieving state-of-the-art performance in detection and tracking. https://github.com/tteepe/TrackTacular