FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection
Not on the map. Why?
Abstract (arXiv)
In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 56 of its 63 references that are in the map. First posted on arXiv 9 Mar 2026.
Builds on
56 of 63 references.
References · 56
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2016Layer Normalizationcited by 5,669
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2015SSD: Single Shot MultiBox Detectorcited by 3,130
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
- 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
- 2024Depth Anything V2cited by 1,422
38 more
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
- 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
- 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
- 2020Center-based 3D Object Detection and Trackingcited by 1,144
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
- 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
- 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
- 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
- 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
- 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
- 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 447
- 2021Deep Long-Tailed Learning: A Surveycited by 417
- 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 400
- 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
- 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
- 2018Rethinking ImageNet Pre-trainingcited by 377
- 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 365
- 2023Scaling Open-Vocabulary Object Detectioncited by 343
- 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
- 2022Vision Transformer with Deformable Attentioncited by 295
- 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
- 2023VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Trackingcited by 222
- 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
- 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
- 2023SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videoscited by 150
- 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
- 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
- 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
- 2022DeepInteraction: 3D Object Detection via Modality Interactioncited by 125
- 2019Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detectioncited by 124
- 2023Cross Modal Transformer: Towards Fast and Robust 3D Object Detectioncited by 70
- 2023Far3D: Expanding the Horizon for Surround-view 3D Object Detectioncited by 70
- 2022Towards Long-Tailed 3D Detectioncited by 40
- 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
- 2023DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Cloudscited by 31
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.