mlsanityData to 7 Oct 2026

PnPNet: End-to-End Perception and Prediction with Tracking in the Loop

arXiv 2005.14711 ↗First posted 29 May 2020cs.CV cs.ROcited by 143

Not on the map. Why?

Cited by 44 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle the problem of joint perception and motion forecasting in the context of self-driving vehicles. Towards this goal we propose PnPNet, an end-to-end model that takes as input sequential sensor data, and outputs at each time step object tracks and their future trajectories. The key component is a novel tracking module that generates object tracks online from detections and exploits trajectory level features for motion forecasting. Specifically, the object tracks get updated at each time step by solving both the data association problem and the trajectory estimation problem. Importantly, the whole model is end-to-end trainable and benefits from joint optimization of all tasks. We validate PnPNet on two large-scale driving datasets, and show significant improvements over the state-of-the-art with better occlusion recovery and more accurate future prediction.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 21; 2023: 29; 2024: 34; 2025: 25; 2026: 13.

143 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  4. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  5. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  6. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  7. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  8. 2023A Language Agent for Autonomous Drivingcited by 161
  9. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  10. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  11. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  12. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
38 more
  1. 2021LaneRCNN: Distributed Representations for Graph-Centric Motion Forecastingcited by 112
  2. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  3. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  4. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  5. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 83
  6. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  7. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  8. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  9. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  10. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  11. 2023Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perceptioncited by 65
  12. 2021Exploring Simple 3D Multi-Object Tracking for Autonomous Drivingcited by 56
  13. 2022Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Cloudscited by 55
  14. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  15. 2022CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusioncited by 50
  16. 2024HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Constructioncited by 48
  17. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  18. 2021Auto4D: Learning to Label 4D Objects from Sequential Point Cloudscited by 42
  19. 2021Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networkscited by 42
  20. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecastingcited by 40
  21. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  22. 2021Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehiclescited by 33
  23. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  24. 2024V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Predictioncited by 31
  25. 2023TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Drivingcited by 30
  26. 2023Implicit Occupancy Flow Fields for Perception and Prediction in Self-Drivingcited by 30
  27. 2022Forecasting from LiDAR via Future Object Detectioncited by 28
  28. 2022Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Predictioncited by 28
  29. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  30. 2023PowerBEV: A Powerful Yet Lightweight Framework for Instance Prediction in Bird's-Eye Viewcited by 24
  31. 2021Deep Multi-Task Learning for Joint Localization, Perception, and Predictioncited by 22
  32. 2021Learning by Watchingcited by 22
  33. 2023Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?cited by 22
  34. 2022End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agentcited by 21
  35. 2022ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Trackingcited by 21
  36. 2023RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloudcited by 21
  37. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?cited by 20
  38. 2021Making Higher Order MOT Scalable: An Efficient Approximate Solver for Lifted Disjoint Pathscited by 19

50 most cited of 143 All 143 (JSON)