# CLIP-BEVFormer: Enhancing Multi-View Image-Based BEV Detector with Ground Truth Flow

arXiv: https://arxiv.org/abs/2403.08919 · HTML: https://mlsanity.com/paper/2403.08919/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision](https://mlsanity.com/paper/2211.10439/) (2022)

## Built on by (most cited first, 14 of 14)

- [WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios](https://mlsanity.com/paper/2510.26125/) (2025)
- [MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation](https://mlsanity.com/paper/2408.09122/) (2024)
- [Feature Map Convergence Evaluation for Functional Module](https://mlsanity.com/paper/2405.04041/) (2024)
- [CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections](https://mlsanity.com/paper/2411.19346/) (2024)
- [Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2502.07631/) (2025)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [DGFusion: Dual-guided Fusion for Robust Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2511.10035/) (2025)
- [MonoDETRNext: Next-Generation Accurate and Efficient Monocular 3D Object Detector](https://mlsanity.com/paper/2405.15176/) (2024)
- [Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model](https://mlsanity.com/paper/2409.11969/) (2024)
- [Hierarchical Feature-level Reverse Propagation for Post-Training Neural Networks](https://mlsanity.com/paper/2506.07188/) (2025)
- [Decoupled Functional Evaluation of Autonomous Driving Models via Feature Map Quality Scoring](https://mlsanity.com/paper/2508.07552/) (2025)
- [Refine-and-Contrast: Adaptive Instance-Aware BEV Representations for Multi-UAV Collaborative Object Detection](https://mlsanity.com/paper/2508.12684/) (2025)
- [StereoMV2D: A Sparse Temporal Stereo-Enhanced Framework for Robust Multi-View 3D Object Detection](https://mlsanity.com/paper/2512.17620/) (2025)
- [Toward Unified Multimodal Representation Learning for Autonomous Driving](https://mlsanity.com/paper/2603.07874/) (2026)

## Abstract (arXiv)

Autonomous driving stands as a pivotal domain in computer vision, shaping the future of transportation. Within this paradigm, the backbone of the system plays a crucial role in interpreting the complex environment. However, a notable challenge has been the loss of clear supervision when it comes to Bird's Eye View elements. To address this limitation, we introduce CLIP-BEVFormer, a novel approach that leverages the power of contrastive learning techniques to enhance the multi-view image-derived BEV backbones with ground truth information flow. We conduct extensive experiments on the challenging nuScenes dataset and showcase significant and consistent improvements over the SOTA. Specifically, CLIP-BEVFormer achieves an impressive 8.5% and 9.2% enhancement in terms of NDS and mAP, respectively, over the previous best BEV model on the 3D object detection task.