# Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding

arXiv: https://arxiv.org/abs/2211.14648 · HTML: https://mlsanity.com/paper/2211.14648/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding](https://mlsanity.com/paper/2105.03655/) (2021)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [Assistive Gym: A Physics Simulation Framework for Assistive Robotics](https://mlsanity.com/paper/1910.04700/) (2019)
- [The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulation](https://mlsanity.com/paper/2002.09093/) (2020)
- [Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors](https://mlsanity.com/paper/2012.00201/) (2020)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Leveraging Multimodal Haptic Sensory Data for Robust Cutting](https://mlsanity.com/paper/1909.12460/) (2019)
- [Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously Unseen Food Items](https://mlsanity.com/paper/1908.07088/) (2019)
- [Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding](https://mlsanity.com/paper/2011.02604/) (2020)
- [Playing with Food: Learning Food Item Representations through Interactive Exploration](https://mlsanity.com/paper/2101.02252/) (2021)
- [Balancing Efficiency and Comfort in Robot-Assisted Bite Transfer](https://mlsanity.com/paper/2111.11401/) (2021)
- [DiSECt: A Differentiable Simulator for Parameter Inference and Control in Robotic Cutting](https://mlsanity.com/paper/2203.10263/) (2022)
- [Learning Semantic Embedding Spaces for Slicing Vegetables](https://mlsanity.com/paper/1904.00303/) (2019)

## Built on by (most cited first, 15 of 15)

- [HYDRA: Hybrid Robot Actions for Imitation Learning](https://mlsanity.com/paper/2306.17237/) (2023)
- [KITE: Keypoint-Conditioned Policies for Semantic Manipulation](https://mlsanity.com/paper/2306.16605/) (2023)
- [Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge](https://mlsanity.com/paper/2411.01796/) (2024)
- [FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes](https://mlsanity.com/paper/2407.07561/) (2024)
- [Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types](https://mlsanity.com/paper/2403.12891/) (2024)
- [FEAST: A Flexible Mealtime-Assistance System Towards In-the-Wild Personalization](https://mlsanity.com/paper/2506.14968/) (2025)
- [Learning Sequential Acquisition Policies for Robot-Assisted Feeding](https://mlsanity.com/paper/2309.05197/) (2023)
- [Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation](https://mlsanity.com/paper/2309.04640/) (2023)
- [IMRL: Integrating Visual, Physical, Temporal, and Geometric Representations for Enhanced Food Acquisition](https://mlsanity.com/paper/2409.12092/) (2024)
- [REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding](https://mlsanity.com/paper/2410.10017/) (2024)
- [Force-Modulated Visual Policy for Robot-Assisted Dressing with Arm Motions](https://mlsanity.com/paper/2509.12741/) (2025)
- [Learning Multimodal Attention for Manipulating Deformable Objects with Changing States](https://mlsanity.com/paper/2309.14837/) (2023)
- [GRITS: A Spillage-Aware Guided Diffusion Policy for Robot Food Scooping Tasks](https://mlsanity.com/paper/2510.00573/) (2025)
- [Robot-Assisted Social Dining as a White Glove Service](https://mlsanity.com/paper/2602.15767/) (2026)
- [VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Models](https://mlsanity.com/paper/2608.18258/) (2026)

## Abstract (arXiv)

Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, datasets, code, and videos are available on our website: https://sites.google.com/view/hapticvisualnet-corl22/home