mlsanityData to 7 Oct 2026

MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception

arXiv 2211.10593 ↗First posted 19 Nov 2022cs.CVcited by 29

On the map: World models → World models for autonomous driving

Abstract (arXiv)

This paper proposes an efficient multi-camera to Bird's-Eye-View (BEV) view transformation method for 3D perception, dubbed MatrixVT. Existing view transformers either suffer from poor transformation efficiency or rely on device-specific operators, hindering the broad application of BEV models. In contrast, our method generates BEV features efficiently with only convolutions and matrix multiplications (MatMul). Specifically, we propose describing the BEV feature as the MatMul of image feature and a sparse Feature Transporting Matrix (FTM). A Prime Extraction module is then introduced to compress the dimension of image features and reduce FTM's sparsity. Moreover, we propose the Ring & Ray Decomposition to replace the FTM with two matrices and reformulate our pipeline to reduce calculation further. Compared to existing methods, MatrixVT enjoys a faster speed and less memory footprint while remaining deploy-friendly. Extensive experiments on the nuScenes benchmark demonstrate that our method is highly efficient but obtains results on par with the SOTA method in object detection and map segmentation tasks

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

32 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  7. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  8. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  9. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  10. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  11. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  12. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
19 more
  1. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  2. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  3. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  4. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  5. 2019Class-balanced Grouping and Sampling for Point Cloud 3D Object Detectioncited by 350
  6. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 340
  7. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  8. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 248
  9. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  10. 2018Orthographic Feature Transform for Monocular 3D Object Detectioncited by 201
  11. 2019CenterMask : Real-Time Anchor-Free Instance Segmentationcited by 200
  12. 2021Probabilistic and Geometric Depth: Detecting Objects in Perspectivecited by 190
  13. 2020Predicting Semantic Map Representations from Images using Pyramid Occupancy Networkscited by 175
  14. 2019Cross-view Semantic Segmentation for Sensing Surroundingscited by 173
  15. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 155
  16. 2022BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereocited by 89
  17. 2020A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye Viewcited by 71
  18. 2022STS: Surround-view Temporal Stereo for Multi-view 3D Detectioncited by 53
  19. 2019Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehiclescited by 18

Built on by

Cited by year

Citing papers per year: 2023: 10; 2024: 14; 2025: 5.

29 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2023FB-BEV: BEV Representation from Forward-Backward View Transformationscited by 98
  3. 2023Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perceptioncited by 65
  4. 2023Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baselinecited by 40
  5. 2023UniWorld: Autonomous Driving Pre-training via World Modelscited by 32
  6. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  7. 2023PointBeV: A Sparse Approach to BeV Predictionscited by 23
  8. 2024GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentationcited by 20
  9. 2023LCPR: A Multi-Scale Attention-Based LiDAR-Camera Fusion Network for Place Recognitioncited by 17
  10. 2024OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networkscited by 17
  11. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  12. 2024InverseMatrixVT3D: An Efficient Projection Matrix-Based Approach for 3D Occupancy Predictioncited by 14
17 more
  1. 2024DualBEV: Unifying Dual View Transformation with Probabilistic Correspondencescited by 9
  2. 2024RayFormer: Improving Query-Based Multi-Camera 3D Object Detection via Ray-Centric Strategiescited by 9
  3. 2023Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentationcited by 8
  4. 2024UCDNet: Multi-UAV Collaborative 3D Object Detection Network by Reliable Feature Mappingcited by 8
  5. 2024Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformationcited by 7
  6. 2024WidthFormer: Toward Efficient Transformer-based BEV View Transformationcited by 5
  7. 2024Panoptic Perception for Autonomous Driving: A Surveycited by 4
  8. 2024LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperationcited by 4
  9. 2023BEV-IO: Enhancing Bird's-Eye-View 3D Detection with Instance Occupancycited by 3
  10. 2025HeightFormer: Learning Height Prediction in Voxel Features for Roadside Vision Centric 3D Object Detection via Transformercited by 3
  11. 2025Coherent Online Road Topology Estimation and Reasoning with Standard-Definition Mapscited by 3
  12. 2023RC-BEVFusion: A Plug-In Module for Radar-Camera Bird's Eye View Feature Fusioncited by 2
  13. 2024Cyclic Refiner: Object-Aware Temporal Representation Learning for Multi-View 3D Detection and Trackingcited by 2
  14. 2025CRAB: Camera-Radar Fusion for Reducing Depth Ambiguity in Backward Projection based View Transformationcited by 2
  15. 2025Fast-BEV++: Fast by Algorithm, Deployable by Designcited by 2
  16. 2024Epipolar Attention Field Transformers for Bird's Eye View Semantic Segmentationcited by 1
  17. 2025DB3D-L: Depth-aware BEV Feature Transformation for Accurate 3D Lane Detectioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021BEV-MODNet: Monocular Camera based Bird's Eye View Moving Object Detection for Autonomous Driving
  2. 2026Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints
  3. 2025MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations
  4. 2023M-BEV: Masked BEV Perception for Robust Autonomous Driving
  5. 2024From Bird's-Eye to Street View: Crafting Diverse and Condition-Aligned Images with Latent Diffusion Model
  6. 2025HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving
  7. 2024OneBEV: Using One Panoramic Image for Bird's-Eye-View Semantic Mapping
  8. 2025BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidance