# Teaching Robots to Build Simulations of Themselves

arXiv: https://arxiv.org/abs/2311.12151 · HTML: https://mlsanity.com/paper/2311.12151/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation](https://mlsanity.com/paper/2204.10850/) (2022)
- [Reconstructing continuous distributions of 3D protein structure from cryo-EM images](https://mlsanity.com/paper/1909.05215/) (2019)
- [NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2307.14620/) (2023)
- [NeRF-RPN: A general framework for object detection in NeRFs](https://mlsanity.com/paper/2211.11646/) (2022)
- [Learning 3D Dynamic Scene Representations for Robot Manipulation](https://mlsanity.com/paper/2011.01968/) (2020)
- [MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection](https://mlsanity.com/paper/2308.09421/) (2023)
- [Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity](https://mlsanity.com/paper/2009.13146/) (2020)
- [Evolving embodied intelligence from materials to machines](https://mlsanity.com/paper/1901.05704/) (2019)
- [On the Origins of Self-Modeling](https://mlsanity.com/paper/2209.02010/) (2022)

## Built on by (most cited first, 6 of 6)

- [AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registration](https://mlsanity.com/paper/2412.05507/) (2024)
- [ArticFlow: Generative Simulation of Articulated Mechanisms](https://mlsanity.com/paper/2511.17883/) (2025)
- [Proprioceptive-visual correspondence enables self-other distinction in humanoid robots](https://mlsanity.com/paper/2606.13222/) (2026)
- [Learning High-Fidelity Robot Self-Model with Articulated 3D Gaussian Splatting](https://mlsanity.com/paper/2503.05398/) (2025)
- [INGRID: Intelligent Generative Robotic Design Using Large Language Models](https://mlsanity.com/paper/2509.03842/) (2025)
- [Localising under the drape: proprioception in the era of distributed surgical robotic system](https://mlsanity.com/paper/2510.23512/) (2025)

## Abstract (arXiv)

The emergence of vision catalysed a pivotal evolutionary advancement, enabling organisms not only to perceive but also to interact intelligently with their environment. This transformation is mirrored by the evolution of robotic systems, where the ability to leverage vision to simulate and predict their own dynamics marks a leap towards autonomy and self-awareness. Humans utilize vision to record experiences and internally simulate potential actions. For example, we can imagine that, if we stand up and raise our arms, the body will form a T shape without physical movement. Similarly, simulation allows robots to plan and predict the outcomes of potential actions without execution. Here we introduce a self-supervised learning framework to enable robots to model and predict their morphology, kinematics and motor control using only brief raw video data, eliminating the need for extensive real-world data collection and kinematic priors. By observing their own movements, akin to humans watching their reflection in a mirror, robots learn an ability to simulate themselves and predict their spatial motion for various tasks. Our results demonstrate that this self-learned simulation not only enables accurate motion planning but also allows the robot to detect abnormalities and recover from damage.