mlsanityData to 14 Oct 2026

MP3: A Unified Model to Map, Perceive, Predict and Plan

arXiv 2101.06806 ↗First posted 18 Jan 2021cs.RO cs.AI cs.CVcited by 212

Not on the map. Why?

Cited by 101 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

High-definition maps (HD maps) are a key component of most modern self-driving systems due to their valuable semantic and geometric information. Unfortunately, building HD maps has proven hard to scale due to their cost as well as the requirements they impose in the localization system that has to work everywhere with centimeter-level accuracy. Being able to drive without an HD map would be very beneficial to scale self-driving solutions as well as to increase the failure tolerance of existing ones (e.g., if localization fails or the map is not up-to-date). Towards this goal, we propose MP3, an end-to-end approach to mapless driving where the input is raw sensor data and a high-level command (e.g., turn left at the intersection). MP3 predicts intermediate representations in the form of an online map and the current and future state of dynamic agents, and exploits them in a novel neural motion planner to make interpretable decisions taking into account uncertainty. We show that our approach is significantly safer, more comfortable, and can follow commands better than the baselines in challenging long-term closed-loop simulations, as well as when compared to an expert driver in a large-scale real-world dataset.

Key facts. Cited by 212 papers in the map (data through 14 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

45 of 58 references.

References · 45

  1. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  2. 2018Group Normalizationcited by 1,524
  3. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  4. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  5. 2016End to End Learning for Self-Driving Carscited by 1,009
  6. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  7. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  8. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  9. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  10. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 307
  11. 2018Learning to Drive in a Daycited by 273
  12. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
33 more
  1. 2021End-to-end Interpretable Neural Motion Plannercited by 242
  2. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  3. 2019Multiple Futures Predictioncited by 178
  4. 2019End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Cloudscited by 171
  5. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 153
  6. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 144
  7. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  8. 2019LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Drivingcited by 133
  9. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  10. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  11. 20183D-LaneNet: End-to-End 3D Multiple Lane Detectioncited by 107
  12. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  13. 2018RoadTracer: Automatic Extraction of Road Networks from Aerial Imagescited by 102
  14. 2020MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Mapscited by 100
  15. 2018Driving Policy Transfer via Modularity and Abstractioncited by 92
  16. 2019Urban Driving with Conditional Imitation Learningcited by 86
  17. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 76
  18. 2020Gen-LaneNet: A Generalized and Scalable Approach for 3D Lane Detectioncited by 75
  19. 2019The Garden of Forking Paths: Towards Multi-Future Trajectory Predictioncited by 72
  20. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  21. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  22. 2020DAGMapper: Learning to Map by Discovering Lane Topologycited by 49
  23. 2020Hierarchical Recurrent Attention Networks for Structured Online Mapscited by 38
  24. 2019Discrete Residual Flow for Probabilistic Pedestrian Behavior Predictioncited by 34
  25. 2017Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labelingcited by 30
  26. 2019Deep Multi-Sensor Lane Detectioncited by 30
  27. 2020The Importance of Prior Knowledge in Precise Multimodal Predictioncited by 25
  28. 2018Search-based optimal motion planning for automated drivingcited by 24
  29. 2019Scene Compliant Trajectory Forecast with Agent-Centric Spatio-Temporal Gridscited by 24
  30. 2020StrObe: Streaming Object Detection from LiDAR Packetscited by 17
  31. 2018An Auto-tuning Framework for Autonomous Vehiclescited by 5
  32. 2019Towards navigation without precise localization: Weakly supervised learning of goal-directed navigation cost mapcited by 4
  33. 2018Learning to Predict Ego-Vehicle Poses for Sampling-Based Nonholonomic Motion Planningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 31; 2023: 59; 2024: 46; 2025: 47; 2026: 16.

212 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  3. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  4. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  6. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  7. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 354
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  9. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  10. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  11. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  12. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 270
38 more
  1. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 248
  2. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  3. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 245
  4. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
  5. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  6. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 241
  7. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  8. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 213
  9. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  10. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  11. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 169
  12. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 166
  13. 2023A Language Agent for Autonomous Drivingcited by 164
  14. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  15. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  16. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 140
  17. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 139
  18. 2022Learning from All Vehiclescited by 135
  19. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  20. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  21. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  22. 2021Learning to drive from a world on railscited by 114
  23. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 104
  24. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  25. 2021Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Imagescited by 98
  26. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 96
  27. 2023Hidden Biases of End-to-End Driving Modelscited by 96
  28. 2022Differentiable Raycasting for Self-supervised Occupancy Forecastingcited by 84
  29. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  30. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecastingcited by 78
  31. 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directionscited by 73
  32. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 73
  33. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policiescited by 69
  34. 2022GoRela: Go Relative for Viewpoint-Invariant Motion Forecastingcited by 68
  35. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  36. 2025Robust Autonomy Emerges from Self-Playcited by 67
  37. 2024Embodied Understanding of Driving Scenarioscited by 63
  38. 2022Occupancy Flow Fields for Motion Forecasting in Autonomous Drivingcited by 62

50 most cited of 212 All 212 (JSON)