# From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence

arXiv: https://arxiv.org/abs/2110.15245 · HTML: https://mlsanity.com/paper/2110.15245/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 191 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 30 of 91)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders](https://mlsanity.com/paper/1802.03480/) (2018)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)

## Built on by (most cited first, 30 of 35)

- [Levels of AGI for Operationalizing Progress on the Path to AGI](https://mlsanity.com/paper/2311.02462/) (2023)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Advances in Embodied Navigation Using Large Language Models: A Survey](https://mlsanity.com/paper/2311.00530/) (2023)
- [From Efficient Multimodal Models to World Models: A Survey](https://mlsanity.com/paper/2407.00118/) (2024)
- [Tree Detection and Diameter Estimation Based on Deep Learning](https://mlsanity.com/paper/2210.17424/) (2022)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos](https://mlsanity.com/paper/2507.16878/) (2025)
- [WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model](https://mlsanity.com/paper/2308.15962/) (2023)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models](https://mlsanity.com/paper/2408.07975/) (2024)
- [TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance](https://mlsanity.com/paper/2509.26627/) (2025)
- [Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning](https://mlsanity.com/paper/2601.03320/) (2026)
- [The Risks of Machine Learning Systems](https://mlsanity.com/paper/2204.09852/) (2022)
- [Invariance is Key to Generalization: Examining the Role of Representation in Sim-to-Real Transfer for Visual Navigation](https://mlsanity.com/paper/2310.15020/) (2023)
- [TongSIM: A General Platform for Simulating Intelligent Machines](https://mlsanity.com/paper/2512.20206/) (2025)
- [Social Neuro AI: Social Interaction as the "dark matter" of AI](https://mlsanity.com/paper/2112.15459/) (2021)
- [Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2206.00270/) (2022)
- [Learning Type-Generalized Actions for Symbolic Planning](https://mlsanity.com/paper/2308.04867/) (2023)
- [Compositional Learning of Visually-Grounded Concepts Using Reinforcement](https://mlsanity.com/paper/2309.04504/) (2023)
- [Ark: An Open-source Python-based Framework for Robot Learning](https://mlsanity.com/paper/2506.21628/) (2025)
- [Multimodal Data Storage and Retrieval for Embodied AI: A Survey](https://mlsanity.com/paper/2508.13901/) (2025)
- [DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning](https://mlsanity.com/paper/2509.17684/) (2025)
- [Enactive Artificial Intelligence: Subverting Gender Norms in Robot-Human Interaction](https://mlsanity.com/paper/2301.08741/) (2023)
- [Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing](https://mlsanity.com/paper/2307.05834/) (2023)
- [Sparse and Transferable Universal Singular Vectors Attack](https://mlsanity.com/paper/2401.14031/) (2024)
- [Bidirectional Progressive Neural Networks with Episodic Return Progress for Emergent Task Sequencing and Robotic Skill Transfer](https://mlsanity.com/paper/2403.04001/) (2024)
- [Embodied Intelligence: The Key to Unblocking Generalized Artificial Intelligence](https://mlsanity.com/paper/2505.06897/) (2025)
- [FlowHijack: A Dynamics-Aware Backdoor Attack on Flow-Matching Vision-Language-Action Models](https://mlsanity.com/paper/2604.09651/) (2026)
- [VEOcc: Voxel-Centric Online Semantic Occupancy Prediction For Embodied Scene Understanding](https://mlsanity.com/paper/2605.25059/) (2026)

## Abstract (arXiv)

Machine learning has long since become a keystone technology, accelerating science and applications in a broad range of domains. Consequently, the notion of applying learning methods to a particular problem set has become an established and valuable modus operandi to advance a particular field. In this article we argue that such an approach does not straightforwardly extended to robotics -- or to embodied intelligence more generally: systems which engage in a purposeful exchange of energy and information with a physical environment. In particular, the purview of embodied intelligent agents extends significantly beyond the typical considerations of main-stream machine learning approaches, which typically (i) do not consider operation under conditions significantly different from those encountered during training; (ii) do not consider the often substantial, long-lasting and potentially safety-critical nature of interactions during learning and deployment; (iii) do not require ready adaptation to novel tasks while at the same time (iv) effectively and efficiently curating and extending their models of the world through targeted and deliberate actions. In reality, therefore, these limitations result in learning-based systems which suffer from many of the same operational shortcomings as more traditional, engineering-based approaches when deployed on a robot outside a well defined, and often narrow operating envelope. Contrary to viewing embodied intelligence as another application domain for machine learning, here we argue that it is in fact a key driver for the advancement of machine learning technology. In this article our goal is to highlight challenges and opportunities that are specific to embodied intelligence and to propose research directions which may significantly advance the state-of-the-art in robot learning.