# REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding

arXiv: https://arxiv.org/abs/2410.10017 · HTML: https://mlsanity.com/paper/2410.10017/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [Visual Grounding of Learned Physical Models](https://mlsanity.com/paper/2004.13664/) (2020)
- [Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo](https://mlsanity.com/paper/2212.00541/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation](https://mlsanity.com/paper/2303.02346/) (2023)
- [LagNetViP: A Lagrangian Neural Network for Video Prediction](https://mlsanity.com/paper/2010.12932/) (2020)
- [Learning Bimanual Scooping Policies for Food Acquisition](https://mlsanity.com/paper/2211.14652/) (2022)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Leveraging Multimodal Haptic Sensory Data for Robust Cutting](https://mlsanity.com/paper/1909.12460/) (2019)
- [Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding](https://mlsanity.com/paper/2211.14648/) (2022)
- [Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously Unseen Food Items](https://mlsanity.com/paper/1908.07088/) (2019)
- [RCareWorld: A Human-centric Simulation World for Caregiving Robots](https://mlsanity.com/paper/2210.10821/) (2022)
- [It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying](https://mlsanity.com/paper/2209.12890/) (2022)
- [Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding](https://mlsanity.com/paper/2011.02604/) (2020)
- [LAVA: Long-horizon Visual Action based Food Acquisition](https://mlsanity.com/paper/2403.12876/) (2024)
- [MORPHeus: a Multimodal One-armed Robot-assisted Peeling System with Human Users In-the-loop](https://mlsanity.com/paper/2404.06570/) (2024)
- [FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes](https://mlsanity.com/paper/2407.07561/) (2024)
- [Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control](https://mlsanity.com/paper/2403.04067/) (2024)
- [Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types](https://mlsanity.com/paper/2403.12891/) (2024)
- [Learning Sequential Acquisition Policies for Robot-Assisted Feeding](https://mlsanity.com/paper/2309.05197/) (2023)
- [Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups](https://mlsanity.com/paper/2207.03348/) (2022)
- [SPARCS: Structuring Physically Assistive Robotics for Caregiving with Stakeholders-in-the-loop](https://mlsanity.com/paper/2210.11022/) (2022)
- [Robot-Assisted Feeding: Generalizing Skewering Strategies across Food Items on a Realistic Plate](https://mlsanity.com/paper/1906.02350/) (2019)

## Built on by (most cited first, 2 of 2)

- [FEAST: A Flexible Mealtime-Assistance System Towards In-the-Wild Personalization](https://mlsanity.com/paper/2506.14968/) (2025)
- [Mitigating Cross-Modal Distraction and Ensuring Geometric Feasibility via Affordance-Guided and Self-Consistent MLLMs for Task Planning in Instruction-Following Manipulation](https://mlsanity.com/paper/2503.13055/) (2025)

## Abstract (arXiv)

The paper presents REPeat, a Real2Sim2Real framework designed to enhance bite acquisition in robot-assisted feeding for soft foods. It uses `pre-acquisition actions' such as pushing, cutting, and flipping to improve the success rate of bite acquisition actions such as skewering, scooping, and twirling. If the data-driven model predicts low success for direct bite acquisition, the system initiates a Real2Sim phase, reconstructing the food's geometry in a simulation. The robot explores various pre-acquisition actions in the simulation, then a Sim2Real step renders a photorealistic image to reassess success rates. If the success improves, the robot applies the action in reality. We evaluate the system on 15 diverse plates with 10 types of food items for a soft food diet, showing improvement in bite acquisition success rates by 27% on average across all plates. See our project website at https://emprise.cs.cornell.edu/repeat.