# VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects

arXiv: https://arxiv.org/abs/2210.03701 · HTML: https://mlsanity.com/paper/2210.03701/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2022.

## Built on by (most cited first, 13 of 13)

- [D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement](https://mlsanity.com/paper/2309.16118/) (2023)
- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation](https://mlsanity.com/paper/2312.13469/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects](https://mlsanity.com/paper/2404.12524/) (2024)
- [Learning Generalizable Tool-use Skills through Trajectory Generation](https://mlsanity.com/paper/2310.00156/) (2023)
- [Built Different: Tactile Perception to Overcome Cross-Embodiment Capability Differences in Collaborative Manipulation](https://mlsanity.com/paper/2409.14896/) (2024)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback](https://mlsanity.com/paper/2305.14470/) (2023)
- [ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation](https://mlsanity.com/paper/2506.12239/) (2025)
- [Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusion](https://mlsanity.com/paper/2410.16571/) (2024)
- [Shape from Shading for Robotic Manipulation](https://mlsanity.com/paper/2304.11824/) (2023)
- [VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds](https://mlsanity.com/paper/2607.16146/) (2026)

## Abstract (arXiv)

Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show:(i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations.