# Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

arXiv: https://arxiv.org/abs/2012.12395 · HTML: https://mlsanity.com/paper/2012.12395/

## Key facts

- Cited by 228 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2020.

## Built on by (most cited first, 30 of 228)

- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Learning Distilled Collaboration Graph for Multi-Agent Perception](https://mlsanity.com/paper/2111.00643/) (2021)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion](https://mlsanity.com/paper/2311.01017/) (2023)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)
- [FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection](https://mlsanity.com/paper/2106.12449/) (2021)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [Space-Time-Separable Graph Convolutional Network for Pose Forecasting](https://mlsanity.com/paper/2110.04573/) (2021)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [3D-MAN: 3D Multi-frame Attention Network for Object Detection](https://mlsanity.com/paper/2103.16054/) (2021)
- [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)

## Abstract (arXiv)

In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor. By jointly reasoning about these tasks, our holistic approach is more robust to occlusion as well as sparse data at range. Our approach performs 3D convolutions across space and time over a bird's eye view representation of the 3D world, which is very efficient in terms of both memory and computation. Our experiments on a new very large scale dataset captured in several north american cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms.