# The Limits and Potentials of Deep Learning for Robotics

arXiv: https://arxiv.org/abs/1804.06557 · HTML: https://mlsanity.com/paper/1804.06557/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2018.

## Built on by (most cited first, 30 of 104)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Learning a Diffusion Model Policy from Rewards via Q-Score Matching](https://mlsanity.com/paper/2312.11752/) (2023)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Robust Reinforcement Learning using Offline Data](https://mlsanity.com/paper/2208.05129/) (2022)
- [WorldGym: World Model as An Environment for Policy Evaluation](https://mlsanity.com/paper/2506.00613/) (2025)
- [Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fields](https://mlsanity.com/paper/2209.08718/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/2303.02783/) (2023)
- [Adversarial Robustness with Semi-Infinite Constrained Learning](https://mlsanity.com/paper/2110.15767/) (2021)
- [Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation](https://mlsanity.com/paper/2307.08875/) (2023)
- [SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection](https://mlsanity.com/paper/2208.13930/) (2022)
- [Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation](https://mlsanity.com/paper/2201.01666/) (2022)
- [Uncertainty for Identifying Open-Set Errors in Visual Object Detection](https://mlsanity.com/paper/2104.01328/) (2021)
- [Bayesian Deep Basis Fitting for Depth Completion with Uncertainty](https://mlsanity.com/paper/2103.15254/) (2021)
- [Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction](https://mlsanity.com/paper/2211.08701/) (2022)
- [Deep Learning for Inertial Positioning: A Survey](https://mlsanity.com/paper/2303.03757/) (2023)
- [Distributionally Robust Policy Learning via Adversarial Environment Generation](https://mlsanity.com/paper/2107.06353/) (2021)
- [Deep Reinforcement Learning-based Obstacle Avoidance for Robot Movement in Warehouse Environments](https://mlsanity.com/paper/2409.14972/) (2024)
- [Sampling from Energy-based Policies using Diffusion](https://mlsanity.com/paper/2410.01312/) (2024)
- [Advances in Inference and Representation for Simultaneous Localization and Mapping](https://mlsanity.com/paper/2103.05041/) (2021)
- [Maximum Likelihood Constraint Inference from Stochastic Demonstrations](https://mlsanity.com/paper/2102.12554/) (2021)
- [Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners](https://mlsanity.com/paper/2406.00430/) (2024)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control](https://mlsanity.com/paper/2104.03893/) (2021)
- [Hyperdimensional Feature Fusion for Out-Of-Distribution Detection](https://mlsanity.com/paper/2112.05341/) (2021)
- [Open-Set Recognition in the Age of Vision-Language Models](https://mlsanity.com/paper/2403.16528/) (2024)
- [Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos](https://mlsanity.com/paper/2409.20557/) (2024)

## Abstract (arXiv)

The application of deep learning in robotics leads to very specific problems and research questions that are typically not addressed by the computer vision and machine learning communities. In this paper we discuss a number of robotics-specific learning, reasoning, and embodiment challenges for deep learning. We explain the need for better evaluation metrics, highlight the importance and unique challenges for deep robotic learning in simulation, and explore the spectrum between purely data-driven and model-driven approaches. We hope this paper provides a motivating overview of important research directions to overcome the current limitations, and help fulfill the promising potentials of deep learning in robotics.