# Bridged Transformer for Vision and Point Cloud 3D Object Detection

arXiv: https://arxiv.org/abs/2210.01391 · HTML: https://mlsanity.com/paper/2210.01391/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [HRFormer: High-Resolution Transformer for Dense Prediction](https://mlsanity.com/paper/2110.09408/) (2021)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020)
- [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://mlsanity.com/paper/2001.10692/) (2020)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)

## Built on by (most cited first, 20 of 20)

- [RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2301.10222/) (2023)
- [C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection](https://mlsanity.com/paper/2306.16175/) (2023)
- [3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2308.16632/) (2023)
- [PanoContext-Former: Panoramic Total Scene Understanding with a Transformer](https://mlsanity.com/paper/2305.12497/) (2023)
- [2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision](https://mlsanity.com/paper/2310.12817/) (2023)
- [A Comprehensive Survey of Transformers for Computer Vision](https://mlsanity.com/paper/2211.06004/) (2022)
- [3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for Indoor 3D Object Detection](https://mlsanity.com/paper/2410.01647/) (2024)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers](https://mlsanity.com/paper/2401.01752/) (2024)
- [Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Survey](https://mlsanity.com/paper/2410.19872/) (2024)
- [CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets](https://mlsanity.com/paper/2302.06148/) (2023)
- [ViPFormer: Efficient Vision-and-Pointcloud Transformer for Unsupervised Pointcloud Understanding](https://mlsanity.com/paper/2303.14376/) (2023)
- [RGB2Point: 3D Point Cloud Generation from Single RGB Images](https://mlsanity.com/paper/2407.14979/) (2024)
- [Efficient Transformer-based 3D Object Detection with Dynamic Token Halting](https://mlsanity.com/paper/2303.05078/) (2023)
- [Position Embedding Needs an Independent Layer Normalization](https://mlsanity.com/paper/2212.05262/) (2022)
- [SVT: Supertoken Video Transformer for Efficient Video Understanding](https://mlsanity.com/paper/2304.00325/) (2023)
- [High-quality Pseudo-labeling for Point Cloud Segmentation with Scene-level Annotation](https://mlsanity.com/paper/2506.23227/) (2025)
- [MedPCFM: Improving Medical Point Cloud Completion by Integrating Point Transformers and Flow Matching](https://mlsanity.com/paper/2606.24433/) (2026)
- [Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience](https://mlsanity.com/paper/2407.16716/) (2024)
- [RYOPO: Bringing End-to-End Category-Level Object Pose Estimation into Real Time](https://mlsanity.com/paper/2610.03013/) (2026)

## Abstract (arXiv)

3D object detection is a crucial research topic in computer vision, which usually uses 3D point clouds as input in conventional setups. Recently, there is a trend of leveraging multiple sources of input data, such as complementing the 3D point cloud with 2D images that often have richer color and fewer noises. However, due to the heterogeneous geometrics of the 2D and 3D representations, it prevents us from applying off-the-shelf neural networks to achieve multimodal fusion. To that end, we propose Bridged Transformer (BrT), an end-to-end architecture for 3D object detection. BrT is simple and effective, which learns to identify 3D and 2D object bounding boxes from both points and image patches. A key element of BrT lies in the utilization of object queries for bridging 3D and 2D spaces, which unifies different sources of data representations in Transformer. We adopt a form of feature aggregation realized by point-to-patch projections which further strengthen the correlations between images and points. Moreover, BrT works seamlessly for fusing the point cloud with multi-view images. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.