# Full-Body Visual Self-Modeling of Robot Morphologies

arXiv: https://arxiv.org/abs/2111.06389 · HTML: https://mlsanity.com/paper/2111.06389/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Emergence of Exploratory Look-Around Behaviors through Active Observation Completion](https://mlsanity.com/paper/1906.11407/) (2019)
- [Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models](https://mlsanity.com/paper/2105.12724/) (2021)

## Built on by (most cited first, 8 of 8)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Differentiable Robot Rendering](https://mlsanity.com/paper/2410.13851/) (2024)
- [Landscape Learning for Neural Network Inversion](https://mlsanity.com/paper/2206.09027/) (2022)
- [Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing](https://mlsanity.com/paper/2210.09297/) (2022)
- [Rotation-Equivariant Conditional Spherical Neural Fields for Learning a Natural Illumination Prior](https://mlsanity.com/paper/2206.03858/) (2022)
- [F-INR: Functional Tensor Decomposition for Implicit Neural Representations](https://mlsanity.com/paper/2503.21507/) (2025)
- [RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior](https://mlsanity.com/paper/2311.09361/) (2023)
- [Evidence of an Emergent "Self" in Continual Robot Learning](https://mlsanity.com/paper/2603.24350/) (2026)

## Abstract (arXiv)

Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These "self-models" allow robots to consider outcomes of multiple possible future actions, without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward-kinema-tics models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics, without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward-kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot's state. Such query-driven self models are continuous in the spatial domain, memory efficient, fully differentiable and kinematic aware. In physical experiments, we demonstrate how a visual self-model is accurate to about one percent of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize and recover from real-world damage, leading to improved machine resiliency. Our project website is at: https://robot-morphology.cs.columbia.edu/