# Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition

arXiv: https://arxiv.org/abs/2210.00843 · HTML: https://mlsanity.com/paper/2210.00843/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects](https://mlsanity.com/paper/1904.03167/) (2019)
- [OmniMAE: Single Model Masked Pretraining on Images and Videos](https://mlsanity.com/paper/2206.08356/) (2022)
- [EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets](https://mlsanity.com/paper/1902.01626/) (2019)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [Memory Efficient Class-Incremental Learning for Image Classification](https://mlsanity.com/paper/2008.01411/) (2020)
- [Online Continual Learning with Contrastive Vision Transformer](https://mlsanity.com/paper/2207.13516/) (2022)
- [Continual Learning with Transformers for Image Classification](https://mlsanity.com/paper/2206.14085/) (2022)
- [Unsupervised Domain Adaptation through Inter-modal Rotation for RGB-D Object Recognition](https://mlsanity.com/paper/2004.10016/) (2020)
- [When CNNs Meet Random RNNs: Towards Multi-Level Analysis for RGB-D Object and Scene Recognition](https://mlsanity.com/paper/2004.12349/) (2020)
- [Recognizing Objects In-the-wild: Where Do We Stand?](https://mlsanity.com/paper/1709.05862/) (2017)
- [Mask-guided Vision Transformer (MG-ViT) for Few-Shot Learning](https://mlsanity.com/paper/2205.09995/) (2022)
- [Recurrent Convolutional Fusion for RGB-D Object Recognition](https://mlsanity.com/paper/1806.01673/) (2018)
- [(DE)^2 CO: Deep Depth Colorization](https://mlsanity.com/paper/1703.10881/) (2017)

## Built on by (most cited first, 12 of 12)

- [Learning the RoPEs: Better 2D and 3D Position Encodings with STRING](https://mlsanity.com/paper/2502.02562/) (2025)
- [ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals](https://mlsanity.com/paper/2309.12312/) (2023)
- [UniMat: Unifying Materials Embeddings through Multi-modal Learning](https://mlsanity.com/paper/2411.08664/) (2024)
- [MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.15209/) (2024)
- [An Investigation of Visual Foundation Models Robustness](https://mlsanity.com/paper/2508.16225/) (2025)
- [Human-Inspired Topological Representations for Visual Object Recognition in Unseen Environments](https://mlsanity.com/paper/2309.08239/) (2023)
- [LM-MCVT: A Lightweight Multi-modal Multi-view Convolutional-Vision Transformer Approach for 3D Object Recognition](https://mlsanity.com/paper/2504.19256/) (2025)
- [DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding](https://mlsanity.com/paper/2603.11380/) (2026)
- [Tri-Modal Fusion Transformers for UAV-based Object Detection](https://mlsanity.com/paper/2604.16630/) (2026)
- [DA-Fusion: Deformable Attention-Based RGB-D Fusion Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2607.17754/) (2026)
- [DINOcular: Self-Supervised Visuospatial Representations](https://mlsanity.com/paper/2608.27226/) (2026)
- [Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?](https://mlsanity.com/paper/2610.17079/) (2026)

## Abstract (arXiv)

The Vision Transformer (ViT) architecture has established its place in computer vision literature, however, training ViTs for RGB-D object recognition remains an understudied topic, viewed in recent literature only through the lens of multi-task pretraining in multiple vision modalities. Such approaches are often computationally intensive, relying on the scale of multiple pretraining datasets to align RGB with 3D information. In this work, we propose a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains for 3D object recognition, focusing on fusing RGB and depth representations encoded jointly by the ViT. Compared to previous works in multimodal Transformers, the key challenge here is to use the attested flexibility of ViTs to capture cross-modal interactions at the downstream and not the pretraining stage. We explore which depth representation is better in terms of resulting accuracy and compare early and late fusion techniques for aligning the RGB and depth modalities within the ViT architecture. Experimental results in the Washington RGB-D Objects dataset (ROD) demonstrate that in such RGB -> RGB-D scenarios, late fusion techniques work better than most popularly employed early fusion. With our transfer baseline, fusion ViTs score up to 95.4% top-1 accuracy in ROD, achieving new state-of-the-art results in this benchmark. We further show the benefits of using our multimodal fusion baseline over unimodal feature extractors in a synthetic-to-real visual adaptation as well as in an open-ended lifelong learning scenario in the ROD benchmark, where our model outperforms previous works by a margin of >8%. Finally, we integrate our method with a robot framework and demonstrate how it can serve as a perception utility in an interactive robot learning scenario, both in simulation and with a real robot.