# Identifying Terrain Physical Parameters from Vision -- Towards Physical-Parameter-Aware Locomotion and Navigation

arXiv: https://arxiv.org/abs/2408.16567 · HTML: https://mlsanity.com/paper/2408.16567/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Fast Traversability Estimation for Wild Visual Navigation](https://mlsanity.com/paper/2305.08510/) (2023)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [Safe Robot Navigation via Multi-Modal Anomaly Detection](https://mlsanity.com/paper/2001.07934/) (2020)
- [How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability](https://mlsanity.com/paper/2209.10788/) (2022)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End](https://mlsanity.com/paper/2310.03581/) (2023)
- [Learning to walk in confined spaces using 3D representation](https://mlsanity.com/paper/2403.00187/) (2024)
- [Learning Agile Locomotion on Risky Terrains](https://mlsanity.com/paper/2311.10484/) (2023)
- [Scientific Exploration of Challenging Planetary Analog Environments with a Team of Legged Robots](https://mlsanity.com/paper/2307.10079/) (2023)
- [Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision](https://mlsanity.com/paper/2404.07110/) (2024)
- [Learning to See Physical Properties with Active Sensing Motor Policies](https://mlsanity.com/paper/2311.01405/) (2023)
- [Seeing Through the Grass: Semantic Pointcloud Filter for Support Surface Learning](https://mlsanity.com/paper/2305.07995/) (2023)
- [Terrain-Aware Quadrupedal Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2310.04675/) (2023)
- [Real-time Digital Double Framework to Predict Collapsible Terrains for Legged Robots](https://mlsanity.com/paper/2209.09508/) (2022)

## Built on by (most cited first, 10 of 10)

- [GaussGym: An open-source real-to-sim framework for learning locomotion from pixels](https://mlsanity.com/paper/2510.15352/) (2025)
- [Learning Humanoid Locomotion with World Model Reconstruction](https://mlsanity.com/paper/2502.16230/) (2025)
- [Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments](https://mlsanity.com/paper/2409.20445/) (2024)
- [Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models](https://mlsanity.com/paper/2410.00231/) (2024)
- [PhysGS: Bayesian-Inferred Gaussian Splatting for Physical Property Estimation](https://mlsanity.com/paper/2511.18570/) (2025)
- [Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving](https://mlsanity.com/paper/2412.00581/) (2024)
- [VLD: Visual Language Goal Distance for Reinforcement Learning Navigation](https://mlsanity.com/paper/2512.07976/) (2025)
- [Learning Impact-Rich Rotational Maneuvers via Centroidal Velocity Rewards and Sim-to-Real Techniques: A One-Leg Hopper Flip Case Study](https://mlsanity.com/paper/2505.12222/) (2025)
- [Beyond Thermal Imaging: Inferring Thermophysical Properties from Time-Resolved Thermal Observations](https://mlsanity.com/paper/2607.07962/) (2026)
- [From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction](https://mlsanity.com/paper/2610.17630/) (2026)

## Abstract (arXiv)

Identifying the physical properties of the surrounding environment is essential for robotic locomotion and navigation to deal with non-geometric hazards, such as slippery and deformable terrains. It would be of great benefit for robots to anticipate these extreme physical properties before contact; however, estimating environmental physical parameters from vision is still an open challenge. Animals can achieve this by using their prior experience and knowledge of what they have seen and how it felt. In this work, we propose a cross-modal self-supervised learning framework for vision-based environmental physical parameter estimation, which paves the way for future physical-property-aware locomotion and navigation. We bridge the gap between existing policies trained in simulation and identification of physical terrain parameters from vision. We propose to train a physical decoder in simulation to predict friction and stiffness from multi-modal input. The trained network allows the labeling of real-world images with physical parameters in a self-supervised manner to further train a visual network during deployment, which can densely predict the friction and stiffness from image data. We validate our physical decoder in simulation and the real world using a quadruped ANYmal robot, outperforming an existing baseline method. We show that our visual network can predict the physical properties in indoor and outdoor experiments while allowing fast adaptation to new environments.