# PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics

arXiv: https://arxiv.org/abs/2410.07688 · HTML: https://mlsanity.com/paper/2410.07688/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)
- [Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics](https://mlsanity.com/paper/2304.14369/) (2023)
- [SpeedFolding: Learning Efficient Bimanual Folding of Garments](https://mlsanity.com/paper/2208.10552/) (2022)
- [PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations](https://mlsanity.com/paper/2404.04421/) (2024)
- [GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion](https://mlsanity.com/paper/2104.05177/) (2021)
- [Differentiable Simulation of Soft Multi-body Systems](https://mlsanity.com/paper/2205.01758/) (2022)
- [Virtual Elastic Objects](https://mlsanity.com/paper/2201.04623/) (2022)
- [Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation](https://mlsanity.com/paper/2206.02881/) (2022)
- [Learning Free-Form Deformations for 3D Object Reconstruction](https://mlsanity.com/paper/1803.10932/) (2018)
- [ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity](https://mlsanity.com/paper/2404.16027/) (2024)
- [Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity](https://mlsanity.com/paper/2009.13146/) (2020)
- [HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects](https://mlsanity.com/paper/2301.07652/) (2023)
- [DefGraspSim: Physics-based simulation of grasp outcomes for 3D deformable objects](https://mlsanity.com/paper/2203.11274/) (2022)
- [3D-MVP: 3D Multiview Pretraining for Robotic Manipulation](https://mlsanity.com/paper/2406.18158/) (2024)
- [TRTM: Template-based Reconstruction and Target-oriented Manipulation of Crumpled Cloths](https://mlsanity.com/paper/2308.04670/) (2023)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [Fast Point Cloud to Mesh Reconstruction for Deformable Object Tracking](https://mlsanity.com/paper/2311.02749/) (2023)

## Built on by (most cited first, 4 of 4)

- [LLM-as-a-Verifier: A General-Purpose Verification Framework](https://mlsanity.com/paper/2607.05391/) (2026)
- [DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation](https://mlsanity.com/paper/2609.23103/) (2026)
- [TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects](https://mlsanity.com/paper/2603.17068/) (2026)
- [Velox: Learning Representations of 4D Geometry and Appearance](https://mlsanity.com/paper/2605.04527/) (2026)

## Abstract (arXiv)

Data-driven methods have shown great potential in solving challenging manipulation tasks; however, their application in the domain of deformable objects has been constrained, in part, by the lack of data. To address this lack, we propose PokeFlex, a dataset featuring real-world multimodal data that is paired and annotated. The modalities include 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks, such as online 3D mesh reconstruction, and it can potentially enable underexplored applications such as the real-world deployment of traditional control methods based on mesh simulations. To deal with the challenges posed by real-world 3D mesh reconstruction, we leverage a professional volumetric capture system that allows complete 360deg reconstruction. PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping objects onto a flat surface or by poking the objects with a robot arm. Interaction wrenches and contact locations are also reported for the latter case. Using different data modalities, we demonstrated a use case for our dataset training models that, given the novelty of the multimodal nature of Pokeflex, constitute the state-of-the-art in multi-object online template-based mesh reconstruction from multimodal data, to the best of our knowledge. We refer the reader to our website ( https://pokeflex-dataset.github.io/ ) for further demos and examples.