# End-to-end Learning of Multi-sensor 3D Tracking by Detection

arXiv: https://arxiv.org/abs/1806.11534 · HTML: https://mlsanity.com/paper/1806.11534/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2018.

## Built on by (most cited first, 25 of 25)

- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks](https://mlsanity.com/paper/2106.09249/) (2021)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Monocular Quasi-Dense 3D Object Tracking](https://mlsanity.com/paper/2103.07351/) (2021)
- [4D Panoptic LiDAR Segmentation](https://mlsanity.com/paper/2102.12472/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud](https://mlsanity.com/paper/2108.05630/) (2021)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking?](https://mlsanity.com/paper/2208.01957/) (2022)
- [Forecasting from LiDAR via Future Object Detection](https://mlsanity.com/paper/2203.16297/) (2022)
- [3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention](https://mlsanity.com/paper/2203.10926/) (2022)
- [Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving](https://mlsanity.com/paper/2108.04602/) (2021)
- [SoK: On the Semantic AI Security in Autonomous Driving](https://mlsanity.com/paper/2203.05314/) (2022)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [Making Higher Order MOT Scalable: An Efficient Approximate Solver for Lifted Disjoint Paths](https://mlsanity.com/paper/2108.10606/) (2021)
- [Learning of Global Objective for Network Flow in Multi-Object Tracking](https://mlsanity.com/paper/2203.16210/) (2022)
- [Multi-Object Tracking and Segmentation via Neural Message Passing](https://mlsanity.com/paper/2207.07454/) (2022)
- [PNAS-MOT: Multi-Modal Object Tracking with Pareto Neural Architecture Search](https://mlsanity.com/paper/2403.15712/) (2024)
- [Sequential Point Clouds: A Survey](https://mlsanity.com/paper/2204.09337/) (2022)
- [Interactive Multi-scale Fusion of 2D and 3D Features for Multi-object Tracking](https://mlsanity.com/paper/2203.16268/) (2022)
- [Privacy-preserving Pedestrian Tracking using Distributed 3D LiDARs](https://mlsanity.com/paper/2303.09915/) (2023)
- [Navigating Threats: A Survey of Physical Adversarial Attacks on LiDAR Perception Systems in Autonomous Vehicles](https://mlsanity.com/paper/2409.20426/) (2024)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [A Pedestrian Detection and Tracking Framework for Autonomous Cars: Efficient Fusion of Camera and LiDAR Data](https://mlsanity.com/paper/2108.12375/) (2021)
- [GOOD: General Optimization-based Fusion for 3D Object Detection via LiDAR-Camera Object Candidates](https://mlsanity.com/paper/2303.09800/) (2023)

## Abstract (arXiv)

In this paper we propose a novel approach to tracking by detection that can exploit both cameras as well as LIDAR data to produce very accurate 3D trajectories. Towards this goal, we formulate the problem as a linear program that can be solved exactly, and learn convolutional networks for detection as well as matching in an end-to-end manner. We evaluate our model in the challenging KITTI dataset and show very competitive results.