# CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

arXiv: https://arxiv.org/abs/2207.02202 · HTML: https://mlsanity.com/paper/2207.02202/

## Key facts

- Cited by 227 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 68 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)

## Built on by (most cited first, 30 of 227)

- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](https://mlsanity.com/paper/2303.07601/) (2023)
- [Robust Collaborative 3D Object Detection in Presence of Pose Errors](https://mlsanity.com/paper/2211.07214/) (2022)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [An Extensible Framework for Open Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2401.13964/) (2024)
- [Collaboration Helps Camera Overtake LiDAR in 3D Detection](https://mlsanity.com/paper/2303.13560/) (2023)
- [SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving](https://mlsanity.com/paper/2306.09001/) (2023)
- [DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving](https://mlsanity.com/paper/2304.01168/) (2023)
- [HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer](https://mlsanity.com/paper/2304.10628/) (2023)
- [Communication-Efficient Collaborative Perception via Information Filling with Codebook](https://mlsanity.com/paper/2405.04966/) (2024)
- [Bridging the Domain Gap for Multi-Agent Perception](https://mlsanity.com/paper/2210.08451/) (2022)
- [Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception](https://mlsanity.com/paper/2307.13929/) (2023)
- [Learning for Vehicle-to-Vehicle Cooperative Perception under Lossy Communication](https://mlsanity.com/paper/2212.08273/) (2022)
- [V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models](https://mlsanity.com/paper/2408.09251/) (2024)
- [STAMP: Scalable Task And Model-agnostic Collaborative Perception](https://mlsanity.com/paper/2501.18616/) (2025)
- [End-to-End Autonomous Driving through V2X Cooperation](https://mlsanity.com/paper/2404.00717/) (2024)
- [CORE: Cooperative Reconstruction for Multi-Agent Perception](https://mlsanity.com/paper/2307.11514/) (2023)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception](https://mlsanity.com/paper/2403.10145/) (2024)
- [Among Us: Adversarially Robust Collaborative Perception by Consensus](https://mlsanity.com/paper/2303.09495/) (2023)
- [Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles](https://mlsanity.com/paper/2402.07635/) (2024)
- [Collaborative Perception for Connected and Autonomous Driving: Challenges, Possible Solutions and Opportunities](https://mlsanity.com/paper/2401.01544/) (2024)
- [Adaptive Feature Fusion for Cooperative Perception using LiDAR Point Clouds](https://mlsanity.com/paper/2208.00116/) (2022)
- [UMC: A Unified Bandwidth-efficient and Multi-resolution based Collaborative Perception Framework](https://mlsanity.com/paper/2303.12400/) (2023)
- [V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception](https://mlsanity.com/paper/2209.13679/) (2022)
- [CoSDH: Communication-Efficient Collaborative Perception via Supply-Demand Awareness and Intermediate-Late Hybridization](https://mlsanity.com/paper/2503.03430/) (2025)

## Abstract (arXiv)

Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems. These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes. Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems, achieving state-of-the-art performance with real-time inference speed. The code is available at https://github.com/DerrickXuNu/CoBEVT.