# ViT-BEVSeg: A Hierarchical Transformer Network for Monocular Birds-Eye-View Segmentation

arXiv: https://arxiv.org/abs/2205.15667 · HTML: https://mlsanity.com/paper/2205.15667/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Orthographic Feature Transform for Monocular 3D Object Detection](https://mlsanity.com/paper/1811.08188/) (2018)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [Cross-view Semantic Segmentation for Sensing Surroundings](https://mlsanity.com/paper/1906.03560/) (2019)
- [Monocular Semantic Occupancy Grid Mapping with Convolutional Variational Encoder-Decoder Networks](https://mlsanity.com/paper/1804.02176/) (2018)
- [Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images](https://mlsanity.com/paper/2108.03227/) (2021)
- [Generalized Object Detection on Fisheye Cameras for Autonomous Driving: Dataset, Representations and Baseline](https://mlsanity.com/paper/2012.02124/) (2020)
- [UnRectDepthNet: Self-Supervised Monocular Depth Estimation using a Generic Framework for Handling Common Camera Distortion Models](https://mlsanity.com/paper/2007.06676/) (2020)
- [AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving](https://mlsanity.com/paper/1901.05808/) (2019)
- [Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving](https://mlsanity.com/paper/1901.03912/) (2019)
- [Occupancy Grids: A Stochastic Spatial Representation for Active Robot Perception](https://mlsanity.com/paper/1304.1098/) (2013)

## Built on by (most cited first, 7 of 7)

- [Street-View Image Generation from a Bird's-Eye View Layout](https://mlsanity.com/paper/2301.04634/) (2023)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers](https://mlsanity.com/paper/2207.06205/) (2022)
- [LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving](https://mlsanity.com/paper/2307.08850/) (2023)
- [Multi-camera Bird's Eye View Perception for Autonomous Driving](https://mlsanity.com/paper/2309.09080/) (2023)
- [SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset](https://mlsanity.com/paper/2502.01894/) (2025)

## Abstract (arXiv)

Generating a detailed near-field perceptual model of the environment is an important and challenging problem in both self-driving vehicles and autonomous mobile robotics. A Bird Eye View (BEV) map, providing a panoptic representation, is a commonly used approach that provides a simplified 2D representation of the vehicle surroundings with accurate semantic level segmentation for many downstream tasks. Current state-of-the art approaches to generate BEV-maps employ a Convolutional Neural Network (CNN) backbone to create feature-maps which are passed through a spatial transformer to project the derived features onto the BEV coordinate frame. In this paper, we evaluate the use of vision transformers (ViT) as a backbone architecture to generate BEV maps. Our network architecture, ViT-BEVSeg, employs standard vision transformers to generate a multi-scale representation of the input image. The resulting representation is then provided as an input to a spatial transformer decoder module which outputs segmentation maps in the BEV grid. We evaluate our approach on the nuScenes dataset demonstrating a considerable improvement in the performance relative to state-of-the-art approaches.