# Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning

arXiv: https://arxiv.org/abs/2412.02119 · HTML: https://mlsanity.com/paper/2412.02119/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects](https://mlsanity.com/paper/2205.05960/) (2022)
- [EDO-Net: Learning Elastic Properties of Deformable Objects from Graph Dynamics](https://mlsanity.com/paper/2209.08996/) (2022)
- [Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods](https://mlsanity.com/paper/2105.12946/) (2021)
- [Understanding Dynamic Tactile Sensing for Liquid Property Estimation](https://mlsanity.com/paper/2205.08771/) (2022)
- [Target-free Extrinsic Calibration of Event-LiDAR Dyad using Edge Correspondences](https://mlsanity.com/paper/2305.04017/) (2023)
- [GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2303.05193/) (2023)
- [Rigid-Soft Interactive Learning for Robust Grasping](https://mlsanity.com/paper/2003.01584/) (2020)
- [One Fling to Goal: Environment-aware Dynamics for Goal-conditioned Fabric Flinging](https://mlsanity.com/paper/2406.14136/) (2024)
- [Polymer-Based Self-Calibrated Optical Fiber Tactile Sensor](https://mlsanity.com/paper/2303.00619/) (2023)
- [A Haptic-Based Proximity Sensing System for Buried Object in Granular Material](https://mlsanity.com/paper/2411.17083/) (2024)

## Built on by (most cited first, 1 of 1)

- [Lattice Boltzmann Model for Learning Real-World Pixel Dynamicity](https://mlsanity.com/paper/2509.16527/) (2025)

## Abstract (arXiv)

Granular materials (GMs) are ubiquitous in daily life. Understanding their properties is also important, especially in agriculture and industry. However, existing works require dedicated measurement equipment and also need large human efforts to handle a large number of particles. In this paper, we introduce a method for estimating the relative values of particle size and density from the video of the interaction with GMs. It is trained on a visuo-haptic learning framework inspired by a contact model, which reveals the strong correlation between GM properties and the visual-haptic data during the probe-dragging in the GMs. After training, the network can map the visual modality well to the haptic signal and implicitly characterize the relative distribution of particle properties in its latent embeddings, as interpreted in that contact model. Therefore, we can analyze GM properties using the trained encoder, and only visual information is needed without extra sensory modalities and human efforts for labeling. The presented GM property estimator has been extensively validated via comparison and ablation experiments. The generalization capability has also been evaluated and a real-world application on the beach is also demonstrated. Experiment videos are available at urlhttps://sites.google.com/view/gmwork/vhlearning .