# LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation

arXiv: https://arxiv.org/abs/2206.13294 · HTML: https://mlsanity.com/paper/2206.13294/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 41 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View](https://mlsanity.com/paper/2005.04078/) (2020)
- [FISHING Net: Future Inference of Semantic Heatmaps In Grids](https://mlsanity.com/paper/2006.09917/) (2020)
- [BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud](https://mlsanity.com/paper/2006.11436/) (2020)
- [Input-level Inductive Biases for 3D Reconstruction](https://mlsanity.com/paper/2112.03243/) (2021)

## Built on by (most cited first, 29 of 29)

- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images](https://mlsanity.com/paper/2401.09413/) (2024)
- [Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving](https://mlsanity.com/paper/2405.17426/) (2024)
- [PointBeV: A Sparse Approach to BeV Predictions](https://mlsanity.com/paper/2312.00703/) (2023)
- [Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?](https://mlsanity.com/paper/2306.09281/) (2023)
- [GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentation](https://mlsanity.com/paper/2407.14108/) (2024)
- [Toward Real-world BEV Perception: Depth Uncertainty Estimation via Gaussian Splatting](https://mlsanity.com/paper/2504.01957/) (2025)
- [HeightFormer: Explicit Height Modeling without Extra Data for Camera-only 3D Object Detection in Bird's Eye View](https://mlsanity.com/paper/2307.13510/) (2023)
- [Style Blind Domain Generalized Semantic Segmentation via Covariance Alignment and Semantic Consistence Contrastive Learning](https://mlsanity.com/paper/2403.06122/) (2024)
- [GaussRender: Learning 3D Occupancy with Gaussian Rendering](https://mlsanity.com/paper/2502.05040/) (2025)
- [OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks](https://mlsanity.com/paper/2404.14027/) (2024)
- [Improving Bird's Eye View Semantic Segmentation by Task Decomposition](https://mlsanity.com/paper/2404.01925/) (2024)
- [NeMO: Neural Map Growing System for Spatiotemporal Fusion in Bird's-Eye-View and BDD-Map Benchmark](https://mlsanity.com/paper/2306.04540/) (2023)
- [Parametric Depth Based Feature Representation Learning for Object Detection and Segmentation in Bird's Eye View](https://mlsanity.com/paper/2307.04106/) (2023)
- [Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.19221/) (2025)
- [Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation](https://mlsanity.com/paper/2308.14525/) (2023)
- [Robust Bird's Eye View Segmentation by Adapting DINOv2](https://mlsanity.com/paper/2409.10228/) (2024)
- [DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception](https://mlsanity.com/paper/2401.08687/) (2024)
- [FipTR: A Simple yet Effective Transformer Framework for Future Instance Prediction in Autonomous Driving](https://mlsanity.com/paper/2404.12867/) (2024)
- [Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers](https://mlsanity.com/paper/2408.12575/) (2024)
- [Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2507.21367/) (2025)
- [OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation](https://mlsanity.com/paper/2407.13137/) (2024)
- [Holistic Parking Slot Detection with Polygon-Shaped Representations](https://mlsanity.com/paper/2310.11629/) (2023)
- [TopView: Vectorising road users in a bird's eye view from uncalibrated street-level imagery with deep learning](https://mlsanity.com/paper/2412.16229/) (2024)
- [Maps for Autonomous Driving: Full-process Survey and Frontiers](https://mlsanity.com/paper/2509.12632/) (2025)
- [Reconstruction Matters: Learning Geometry-Aligned BEV Representation through 3D Gaussian Splatting](https://mlsanity.com/paper/2603.19193/) (2026)
- [Zero-BEV: Zero-shot Projection of Any First-Person Modality to BEV Maps](https://mlsanity.com/paper/2402.13848/) (2024)
- [BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autonomous Driving](https://mlsanity.com/paper/2408.16322/) (2024)
- [Variational Inference for Bird's Eye View Segmentation in Autonomous Driving](https://mlsanity.com/paper/2607.14710/) (2026)

## Abstract (arXiv)

Recent works in autonomous driving have widely adopted the bird's-eye-view (BEV) semantic map as an intermediate representation of the world. Online prediction of these BEV maps involves non-trivial operations such as multi-camera data extraction as well as fusion and projection into a common topview grid. This is usually done with error-prone geometric operations (e.g., homography or back-projection from monocular depth estimation) or expensive direct dense mapping between image pixels and pixels in BEV (e.g., with MLP or attention). In this work, we present 'LaRa', an efficient encoder-decoder, transformer-based model for vehicle semantic segmentation from multiple cameras. Our approach uses a system of cross-attention to aggregate information over multiple sensors into a compact, yet rich, collection of latent representations. These latent representations, after being processed by a series of self-attention blocks, are then reprojected with a second cross-attention in the BEV space. We demonstrate that our model outperforms the best previous works using transformers on nuScenes. The code and trained models are available at https://github.com/valeoai/LaRa