mlsanityData to 7 Oct 2026

ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries

arXiv 2208.01582 ↗First posted 2 Aug 2022cs.CV cs.ROcited by 116

Not on the map. Why?

Cited by 46 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Perception and prediction are two separate modules in the existing autonomous driving systems. They interact with each other via hand-picked features such as agent bounding boxes and trajectories. Due to this separation, prediction, as a downstream module, only receives limited information from the perception module. To make matters worse, errors from the perception modules can propagate and accumulate, adversely affecting the prediction results. In this work, we propose ViP3D, a query-based visual trajectory prediction pipeline that exploits rich information from raw videos to directly predict future trajectories of agents in a scene. ViP3D employs sparse agent queries to detect, track, and predict throughout the pipeline, making it the first fully differentiable vision-based trajectory prediction approach. Instead of using historical feature maps and trajectories, useful information from previous timestamps is encoded in agent queries, which makes ViP3D a concise streaming prediction method. Furthermore, extensive experimental results on the nuScenes dataset show the strong vision-based prediction performance of ViP3D over traditional pipelines and previous end-to-end models.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 59 of its 72 references that are in the map. First posted on arXiv 2 Aug 2022.

Builds on

59 of 72 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  7. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  8. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  9. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  10. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  12. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
38 more
  1. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  2. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  3. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  4. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  5. 2020Tracking Objects as Pointscited by 483
  6. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  7. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  8. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  9. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  10. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  11. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  12. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  13. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  14. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  15. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  16. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  17. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  18. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  19. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  20. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  21. 2019Disentangling Monocular 3D Object Detectioncited by 215
  22. 2019Multiple Futures Predictioncited by 178
  23. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  24. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  25. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  26. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  27. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  28. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  29. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  30. 2021SimpleTrack: Understanding and Rethinking 3D Multi-object Trackingcited by 102
  31. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  32. 2016Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypothesescited by 96
  33. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 90
  34. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  35. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  36. 2021Monocular Quasi-Dense 3D Object Trackingcited by 80
  37. 2018Multi-Modal Trajectory Prediction of Surrounding Vehicles with Maneuver based LSTMscited by 77
  38. 2021LaPred: Lane-Aware Prediction of Multi-Modal Future Trajectories of Dynamic Agentscited by 64

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 23; 2024: 39; 2025: 32; 2026: 19.

116 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  4. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  5. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  6. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  7. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  8. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  9. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  10. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  11. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  12. 2023Language Prompt for Autonomous Drivingcited by 110
38 more
  1. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  2. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 97
  3. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  4. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  5. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 72
  6. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  7. 2024MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mappingcited by 67
  8. 2023Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perceptioncited by 65
  9. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  10. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  11. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  12. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
  13. 2023Online Vectorized HD Map Construction using Geometrycited by 42
  14. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41
  15. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 35
  16. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  17. 2023PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Drivingcited by 33
  18. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  19. 2024V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Predictioncited by 31
  20. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Drivingcited by 31
  21. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  22. 2022Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Predictioncited by 28
  23. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25
  24. 2023PointBeV: A Sparse Approach to BeV Predictionscited by 23
  25. 2023Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?cited by 22
  26. 2024Adapting to Length Shift: FlexiLength Network for Trajectory Predictioncited by 22
  27. 2024T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memorycited by 20
  28. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?cited by 20
  29. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  30. 2025Toward Real-world BEV Perception: Depth Uncertainty Estimation via Gaussian Splattingcited by 20
  31. 2024Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sportscited by 19
  32. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  33. 2024DualAD: Disentangling the Dynamic and Static World for End-to-End Drivingcited by 17
  34. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 17
  35. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  36. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16
  37. 2025Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directionscited by 14
  38. 2023S.T.A.R.-Track: Latent Motion Models for End-to-End 3D Object Tracking with Adaptive Spatio-Temporal Appearance Representationscited by 12

50 most cited of 116 All 116 (JSON)