# X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation

arXiv: https://arxiv.org/abs/2210.06778 · HTML: https://mlsanity.com/paper/2210.06778/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 76 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)

## Built on by (most cited first, 30 of 38)

- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [DDP: Diffusion Model for Dense Visual Prediction](https://mlsanity.com/paper/2303.17559/) (2023)
- [An Extensible Framework for Open Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2401.13964/) (2024)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)
- [UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalities](https://mlsanity.com/paper/2309.14516/) (2023)
- [X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detection](https://mlsanity.com/paper/2303.02203/) (2023)
- [MapDistill: Boosting Efficient Camera-based HD Map Construction via Camera-LiDAR Fusion Model Distillation](https://mlsanity.com/paper/2407.11682/) (2024)
- [Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation](https://mlsanity.com/paper/2407.11344/) (2024)
- [MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction](https://mlsanity.com/paper/2502.04377/) (2025)
- [4D Panoptic Segmentation as Invariant and Equivariant Field Prediction](https://mlsanity.com/paper/2303.15651/) (2023)
- [MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentation](https://mlsanity.com/paper/2304.09801/) (2023)
- [UniM2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving](https://mlsanity.com/paper/2308.10421/) (2023)
- [MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection](https://mlsanity.com/paper/2412.16876/) (2024)
- [MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation](https://mlsanity.com/paper/2408.09122/) (2024)
- [MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2503.06700/) (2025)
- [M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.18100/) (2025)
- [Navya3DSeg -- Navya 3D Semantic Segmentation Dataset & split generation for autonomous vehicles](https://mlsanity.com/paper/2302.08292/) (2023)
- [DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction](https://mlsanity.com/paper/2303.01573/) (2023)
- [PosSAM: Panoptic Open-vocabulary Segment Anything](https://mlsanity.com/paper/2403.09620/) (2024)
- [Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization](https://mlsanity.com/paper/2505.06635/) (2025)
- [OneBEV: Using One Panoramic Image for Bird's-Eye-View Semantic Mapping](https://mlsanity.com/paper/2409.13912/) (2024)
- [ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment](https://mlsanity.com/paper/2503.02505/) (2025)
- [PCT: Perspective Cue Training Framework for Multi-Camera BEV Segmentation](https://mlsanity.com/paper/2403.12530/) (2024)
- [RopeBEV: A Multi-Camera Roadside Perception Network in Bird's-Eye-View](https://mlsanity.com/paper/2409.11706/) (2024)
- [BroadBEV: Collaborative LiDAR-camera Fusion for Broad-sighted Bird's Eye View Map Construction](https://mlsanity.com/paper/2309.11119/) (2023)
- [LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving](https://mlsanity.com/paper/2307.08850/) (2023)
- [Multi-camera Bird's Eye View Perception for Autonomous Driving](https://mlsanity.com/paper/2309.09080/) (2023)
- [DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning](https://mlsanity.com/paper/2404.06352/) (2024)
- [VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization](https://mlsanity.com/paper/2411.01618/) (2024)
- [Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2505.04619/) (2025)

## Abstract (arXiv)

Bird's-eye-view (BEV) grid is a common representation for the perception of road components, e.g., drivable area, in autonomous driving. Most existing approaches rely on cameras only to perform segmentation in BEV space, which is fundamentally constrained by the absence of reliable depth information. Latest works leverage both camera and LiDAR modalities, but sub-optimally fuse their features using simple, concatenation-based mechanisms. In this paper, we address these problems by enhancing the alignment of the unimodal features in order to aid feature fusion, as well as enhancing the alignment between the cameras' perspective view (PV) and BEV representations. We propose X-Align, a novel end-to-end cross-modal and cross-view learning framework for BEV segmentation consisting of the following components: (i) a novel Cross-Modal Feature Alignment (X-FA) loss, (ii) an attention-based Cross-Modal Feature Fusion (X-FF) module to align multi-modal BEV features implicitly, and (iii) an auxiliary PV segmentation branch with Cross-View Segmentation Alignment (X-SA) losses to improve the PV-to-BEV transformation. We evaluate our proposed method across two commonly used benchmark datasets, i.e., nuScenes and KITTI-360. Notably, X-Align significantly outperforms the state-of-the-art by 3 absolute mIoU points on nuScenes. We also provide extensive ablation studies to demonstrate the effectiveness of the individual components.