mlsanityData to 7 Oct 2026

Deep Multi-Task Learning for Joint Localization, Perception, and Prediction

arXiv 2101.06720 ↗First posted 17 Jan 2021cs.CV cs.LG cs.ROcited by 22

Not on the map. Why?

Abstract (arXiv)

Over the last few years, we have witnessed tremendous progress on many subtasks of autonomous driving, including perception, motion forecasting, and motion planning. However, these systems often assume that the car is accurately localized against a high-definition map. In this paper we question this assumption, and investigate the issues that arise in state-of-the-art autonomy stacks under localization error. Based on our observations, we design a system that jointly performs perception, prediction, and localization. Our architecture is able to reuse computation between both tasks, and is thus able to correct localization errors efficiently. We show experiments on a large-scale autonomy dataset, demonstrating the efficiency and accuracy of our proposed approach.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 56 of its 77 references that are in the map. First posted on arXiv 17 Jan 2021.

Builds on

56 of 77 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  8. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  9. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 556
  10. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  11. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  12. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
38 more
  1. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  2. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  3. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  4. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  5. 2018Learning to Drive in a Daycited by 266
  6. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  7. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  8. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  9. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  10. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  11. 2015Instance-aware Semantic Segmentation via Multi-task Network Cascadescited by 120
  12. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  13. 2016MultiNet: Real-time Joint Semantic Reasoning for Autonomous Drivingcited by 90
  14. 2017One Model To Learn Them Allcited by 86
  15. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  16. 2019DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registrationcited by 77
  17. 2019Does computer vision matter for action?cited by 64
  18. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  19. 2018VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometrycited by 61
  20. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  21. 2018Modular meta-learningcited by 51
  22. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  23. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  24. 2020Learning to Evaluate Perception Models Using Planner-Centric Metricscited by 49
  25. 2020DH3D: Deep Hierarchical 3D Descriptors for Robust Large-Scale 6DoF Relocalizationcited by 42
  26. 2018FutureMapping: The Computational Structure of Spatial AI Systemscited by 35
  27. 2020Learning to Localize Using a LiDAR Intensity Mapcited by 28
  28. 2020The Importance of Prior Knowledge in Precise Multimodal Predictioncited by 25
  29. 2017Robust and Precise Vehicle Localization based on Multi-sensor Fusion in Diverse City Scenescited by 24
  30. 2019Exploiting Sparse Semantic HD Maps for Self-Driving Vehicle Localizationcited by 24
  31. 2017Generating Long-term Trajectories Using Deep Hierarchical Networkscited by 23
  32. 2020STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Predictioncited by 22
  33. 2020RSL-Net: Localising in Satellite Images From a Radar on the Groundcited by 18
  34. 2020Streaming Object Detection for 3-D Point Cloudscited by 17
  35. 2020StrObe: Streaming Object Detection from LiDAR Packetscited by 17
  36. 2019Side-Tuning: A Baseline for Network Adaptation via Additive Side Networkscited by 16
  37. 2019Masking by Moving: Learning Distraction-Free Radar Odometry from Pose Informationcited by 15
  38. 2020Unifying Deep Local and Global Features for Image Searchcited by 12

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 8; 2024: 2; 2025: 3.

22 papers.

Sort:

By area:

  1. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  2. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  3. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  4. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  5. 2023Does Physical Adversarial Example Really Matter to Autonomous Driving? Towards System-Level Effect of Adversarial Object Evasion Attackcited by 37
  6. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  7. 2023TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Drivingcited by 30
  8. 2023DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Predictioncited by 27
  9. 2022LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Predictioncited by 19
  10. 2021Reimagining an autonomous vehiclecited by 11
  11. 2022JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenescited by 11
  12. 2021Multi-Task Neural Processescited by 7
10 more
  1. 2023DELO: Deep Evidential LiDAR Odometry using Partial Optimal Transportcited by 7
  2. 2024Joint Perception and Prediction for Autonomous Driving: A Surveycited by 7
  3. 2024Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signalscited by 5
  4. 2025SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generationcited by 5
  5. 2023DeepSTEP -- Deep Learning-Based Spatio-Temporal End-To-End Perception for Autonomous Vehiclescited by 1
  6. 2025SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Datasetcited by 1
  7. 2025To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Softwarecited by 1
  8. 2022On Steering Multi-Annotations per Sample for Multi-Task Learningcited by 0
  9. 2023Structured Pruning for Multi-Task Deep Neural Networkscited by 0
  10. 2023Locking On: Leveraging Dynamic Vehicle-Imposed Motion Constraints to Improve Visual Localizationcited by 0