# Depth Matters: Multimodal RGB-D Perception for Robust Autonomous Agents

arXiv: https://arxiv.org/abs/2503.16711 · HTML: https://mlsanity.com/paper/2503.16711/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [Depth-Adapted CNN for RGB-D cameras](https://mlsanity.com/paper/2009.09976/) (2020)
- [VisualBackProp: efficient visualization of CNNs](https://mlsanity.com/paper/1611.05418/) (2016)
- [Liquid Resistance Liquid Capacitance Networks](https://mlsanity.com/paper/2403.08791/) (2024)
- [Bypassing the Simulation-to-reality Gap: Online Reinforcement Learning using a Supervisor](https://mlsanity.com/paper/2209.11082/) (2022)
- [TinyLidarNet: 2D LiDAR-based End-to-End Deep Learning Model for F1TENTH Autonomous Racing](https://mlsanity.com/paper/2410.07447/) (2024)
- [Learning with Chemical versus Electrical Synapses -- Does it Make a Difference?](https://mlsanity.com/paper/2401.08602/) (2023)
- [Assessing the Robustness of LiDAR, Radar and Depth Cameras Against Ill-Reflecting Surfaces in Autonomous Vehicles: An Experimental Study](https://mlsanity.com/paper/2309.10504/) (2023)
- [Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering](https://mlsanity.com/paper/2409.12716/) (2024)

## Abstract (arXiv)

Autonomous agents that rely purely on perception to make real-time control decisions require efficient and robust architectures. In this work, we demonstrate that augmenting RGB input with depth information significantly enhances our agents' ability to predict steering commands compared to using RGB alone. We benchmark lightweight recurrent controllers that leverage the fused RGB-D features for sequential decision-making. To train our models, we collect high-quality data using a small-scale autonomous car controlled by an expert driver via a physical steering wheel, capturing varying levels of steering difficulty. Our models were successfully deployed on real hardware and inherently avoided dynamic and static obstacles, under out-of-distribution conditions. Specifically, our findings reveal that the early fusion of depth data results in a highly robust controller, which remains effective even with frame drops and increased noise levels, without compromising the network's focus on the task.