mlsanityData to 14 Oct 2026

TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving

arXiv 2303.09998 ↗First posted 17 Mar 2023cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Vision-centric joint perception and prediction (PnP) has become an emerging trend in autonomous driving research. It predicts the future states of the traffic participants in the surrounding environment from raw RGB images. However, it is still a critical challenge to synchronize features obtained at multiple camera views and timestamps due to inevitable geometric distortions and further exploit those spatial-temporal features. To address this issue, we propose a temporal bird's-eye-view pyramid transformer (TBP-Former) for vision-centric PnP, which includes two novel designs. First, a pose-synchronized BEV encoder is proposed to map raw image inputs with any camera pose at any time to a shared and synchronized BEV space for better spatial-temporal synchronization. Second, a spatial-temporal pyramid transformer is introduced to comprehensively extract multi-scale BEV features and predict future BEV states with the support of spatial-temporal priors. Extensive experiments on nuScenes dataset show that our proposed framework overall outperforms all state-of-the-art vision-based prediction methods.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 51 of its 58 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

51 of 58 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  9. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  10. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  11. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 352
  12. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
38 more
  1. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 281
  2. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  3. 2022Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Mapscited by 255
  4. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 242
  5. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  6. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  7. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 212
  8. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  9. 2019Scaling Autoregressive Video Modelscited by 190
  10. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  11. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 174
  12. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  13. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 156
  14. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 153
  15. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 144
  16. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  17. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  18. 2022GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoningcited by 128
  19. 2021Video Frame Interpolation Transformercited by 123
  20. 2022BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigscited by 123
  21. 2021Translating Images into Mapscited by 115
  22. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  23. 2020Latent Video Transformercited by 106
  24. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  25. 2020MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Mapscited by 100
  26. 2018Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networkscited by 84
  27. 2019Identifying Unknown Instances for Autonomous Drivingcited by 74
  28. 20203D Point Cloud Processing and Learning for Autonomous Drivingcited by 68
  29. 2020Collaborative Motion Prediction via Neural Motion Message Passingcited by 55
  30. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  31. 2021Self-Supervised Pillar Motion Learning for Autonomous Drivingcited by 44
  32. 2022RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolutioncited by 44
  33. 2022Time3D: End-to-End Joint Monocular 3D Object Detection and Tracking for Autonomous Drivingcited by 30
  34. 2021Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillationcited by 25
  35. 2021Deep Multi-Task Learning for Joint Localization, Perception, and Predictioncited by 22
  36. 2020LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusioncited by 18
  37. 2020PillarFlow: End-to-end Birds-eye-view Flow Estimation for Autonomous Drivingcited by 16
  38. 2020Dynamic Occupancy Grid Mapping with Recurrent Neural Networkscited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 8; 2026: 3.

30 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 260
  2. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  3. 2024GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splattingcited by 50
  4. 2024Leveraging Enhanced Queries of Point Sets for Vectorized Map Constructioncited by 46
  5. 2024Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehiclescited by 33
  6. 2023PointBeV: A Sparse Approach to BeV Predictionscited by 23
  7. 2024GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentationcited by 20
  8. 2023HeightFormer: Explicit Height Modeling without Extra Data for Camera-only 3D Object Detection in Bird's Eye Viewcited by 19
  9. 2023Joint-Relation Transformer for Multi-Person Motion Predictioncited by 16
  10. 2025OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with Observer-Forecaster-Refiner Frameworkcited by 12
  11. 2024DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analyticscited by 8
  12. 2024Joint Perception and Prediction for Autonomous Driving: A Surveycited by 7
18 more
  1. 2023Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Drivingcited by 6
  2. 2024TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregationcited by 5
  3. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  4. 2024FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Drivingcited by 4
  5. 2024Progressive Query Refinement Framework for Bird's-Eye-View Semantic Segmentation from Surrounding Imagescited by 2
  6. 2024Fast and Efficient Transformer-based Method for Bird's Eye View Instance Predictioncited by 2
  7. 2024Learning Temporal Cues by Predicting Objects Move for Multi-camera 3D Object Detectioncited by 1
  8. 2025To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Softwarecited by 1
  9. 2025LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptationcited by 1
  10. 2026MASAR: Motion-Appearance Synergy Refinement for Joint Detection and Trajectory Forecastingcited by 1
  11. 2024Relation Learning and Aggregate-attention for Multi-person Motion Predictioncited by 0
  12. 2024UnityGraph: Unified Learning of Spatio-temporal features for Multi-person Motion Predictioncited by 0
  13. 2024HSDA: High-frequency Shuffle Data Augmentation for Bird's-Eye-View Map Segmentationcited by 0
  14. 2025BEVMOSNet: Multimodal Fusion for BEV Moving Object Segmentationcited by 0
  15. 2025BEVTraj: Map-Free End-to-End Trajectory Prediction in Bird's-Eye View with Deformable Attention and Sparse Goal Proposalscited by 0
  16. 2025Spatiotemporal-Untrammelled Mixture of Experts for Multi-Person Motion Predictioncited by 0
  17. 2026CycleBEV: Regularizing View Transformation Networks via View Cycle Consistency for Bird's-Eye-View Semantic Segmentationcited by 0
  18. 2026BEV-Denoise: Learning Intrinsic Noise for Accurate Bird's-Eye-View Semantic Segmentationcited by 0