# Multimodal Deep Learning for Robust RGB-D Object Recognition

arXiv: https://arxiv.org/abs/1507.06821 · HTML: https://mlsanity.com/paper/1507.06821/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2015.

## Built on by (most cited first, 30 of 57)

- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [The Ingredients for Robotic Diffusion Transformers](https://mlsanity.com/paper/2410.10088/) (2024)
- [Specificity-preserving RGB-D Saliency Detection](https://mlsanity.com/paper/2108.08162/) (2021)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [Weakly Aligned Feature Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2204.09848/) (2022)
- [An evidential classifier based on Dempster-Shafer theory and deep learning](https://mlsanity.com/paper/2103.13549/) (2021)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [ObjectMatch: Robust Registration using Canonical Object Correspondences](https://mlsanity.com/paper/2212.01985/) (2022)
- [Structure Flow-Guided Network for Real Depth Super-Resolution](https://mlsanity.com/paper/2301.13416/) (2023)
- [Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition](https://mlsanity.com/paper/2210.00843/) (2022)
- [Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis](https://mlsanity.com/paper/2303.05473/) (2023)
- [DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning](https://mlsanity.com/paper/2109.12065/) (2021)
- [Visual Modality Prompt for Adapting Vision-Language Object Detectors](https://mlsanity.com/paper/2412.00622/) (2024)
- [AFFIRM: Affinity Fusion-based Framework for Iteratively Random Motion correction of multi-slice fetal brain MRI](https://mlsanity.com/paper/2205.05851/) (2022)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection](https://mlsanity.com/paper/2404.18849/) (2024)
- [Illumination and Temperature-Aware Multispectral Networks for Edge-Computing-Enabled Pedestrian Detection](https://mlsanity.com/paper/2112.05053/) (2021)
- [Enhancing image captioning with depth information using a Transformer-based framework](https://mlsanity.com/paper/2308.03767/) (2023)
- [MVIP -- A Dataset and Methods for Application Oriented Multi-View and Multi-Modal Industrial Part Recognition](https://mlsanity.com/paper/2502.15448/) (2025)
- [Mushrooms Detection, Localization and 3D Pose Estimation using RGB-D Sensor for Robotic-picking Applications](https://mlsanity.com/paper/2201.02837/) (2022)
- [Multimodal Side-Tuning for Document Classification](https://mlsanity.com/paper/2301.07502/) (2023)
- [Hardening RGB-D Object Recognition Systems against Adversarial Patch Attacks](https://mlsanity.com/paper/2309.07106/) (2023)
- [Object-Oriented Material Classification and 3D Clustering for Improved Semantic Perception and Mapping in Mobile Robots](https://mlsanity.com/paper/2407.06077/) (2024)
- [Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech](https://mlsanity.com/paper/2112.13758/) (2021)
- [Improving Lidar-Based Semantic Segmentation of Top-View Grid Maps by Learning Features in Complementary Representations](https://mlsanity.com/paper/2203.01151/) (2022)
- [Does depth estimation help object detection?](https://mlsanity.com/paper/2204.06512/) (2022)
- [Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning](https://mlsanity.com/paper/2305.11347/) (2023)
- [Temporal Shift -- Multi-Objective Loss Function for Improved Anomaly Fall Detection](https://mlsanity.com/paper/2311.02863/) (2023)
- [MLMT-CNN for Object Detection and Segmentation in Multi-layer and Multi-spectral Images](https://mlsanity.com/paper/2407.14473/) (2024)
- [ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources](https://mlsanity.com/paper/2502.07862/) (2025)

## Abstract (arXiv)

Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications. This paper leverages recent progress on Convolutional Neural Networks (CNNs) and proposes a novel RGB-D architecture for object recognition. Our architecture is composed of two separate CNN processing streams - one for each modality - which are consecutively combined with a late fusion network. We focus on learning with imperfect sensor data, a typical problem in real-world robotics tasks. For accurate learning, we introduce a multi-stage training methodology and two crucial ingredients for handling depth data with CNNs. The first, an effective encoding of depth information for CNNs that enables learning without the need for large depth datasets. The second, a data augmentation scheme for robust learning with depth images by corrupting them with realistic noise patterns. We present state-of-the-art results on the RGB-D object dataset and show recognition in challenging RGB-D real-world noisy settings.