# Playing with Food: Learning Food Item Representations through Interactive Exploration

arXiv: https://arxiv.org/abs/2101.02252 · HTML: https://mlsanity.com/paper/2101.02252/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2021.

## Built on by (most cited first, 8 of 8)

- [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding](https://mlsanity.com/paper/2211.14648/) (2022)
- [Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions](https://mlsanity.com/paper/2502.15336/) (2025)
- [Visuo-Tactile Zero-Shot Object Recognition with Vision-Language Model](https://mlsanity.com/paper/2409.09276/) (2024)
- [CulinaryCut-VLAP: A Vision-Language-Action-Physics Framework for Food Cutting via a Force-Aware Material Point Method](https://mlsanity.com/paper/2601.06451/) (2026)
- [WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systems](https://mlsanity.com/paper/2603.14392/) (2026)

## Abstract (arXiv)

A key challenge in robotic food manipulation is modeling the material properties of diverse and deformable food items. We propose using a multimodal sensory approach to interact and play with food that facilitates the ability to distinguish these properties across food items. First, we use a robotic arm and an array of sensors, which are synchronized using ROS, to collect a diverse dataset consisting of 21 unique food items with varying slices and properties. Afterwards, we learn visual embedding networks that utilize a combination of proprioceptive, audio, and visual data to encode similarities among food items using a triplet loss formulation. Our evaluations show that embeddings learned through interactions can successfully increase performance in a wide range of material and shape classification tasks. We envision that these learned embeddings can be utilized as a basis for planning and selecting optimal parameters for more material-aware robotic food manipulation skills. Furthermore, we hope to stimulate further innovations in the field of food robotics by sharing this food playing dataset with the research community.