# Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types

arXiv: https://arxiv.org/abs/2403.12891 · HTML: https://mlsanity.com/paper/2403.12891/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Visual Imitation Made Easy](https://mlsanity.com/paper/2008.04899/) (2020)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [An Empirical Study of Spatial Attention Mechanisms in Deep Networks](https://mlsanity.com/paper/1904.05873/) (2019)
- [Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.10190/) (2020)
- [Attention in Attention Network for Image Super-Resolution](https://mlsanity.com/paper/2104.09497/) (2021)
- [Learning Bimanual Scooping Policies for Food Acquisition](https://mlsanity.com/paper/2211.14652/) (2022)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding](https://mlsanity.com/paper/2211.14648/) (2022)
- [Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously Unseen Food Items](https://mlsanity.com/paper/1908.07088/) (2019)
- [Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding](https://mlsanity.com/paper/2011.02604/) (2020)
- [Towards Assistive Feeding with a General-Purpose Mobile Manipulator](https://mlsanity.com/paper/1605.07996/) (2016)

## Built on by (most cited first, 8 of 8)

- [CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems](https://mlsanity.com/paper/2502.17821/) (2025)
- [IMRL: Integrating Visual, Physical, Temporal, and Geometric Representations for Enhanced Food Acquisition](https://mlsanity.com/paper/2409.12092/) (2024)
- [REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding](https://mlsanity.com/paper/2410.10017/) (2024)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation](https://mlsanity.com/paper/2509.18198/) (2025)
- [GRITS: A Spillage-Aware Guided Diffusion Policy for Robot Food Scooping Tasks](https://mlsanity.com/paper/2510.00573/) (2025)
- [Robot-Assisted Social Dining as a White Glove Service](https://mlsanity.com/paper/2602.15767/) (2026)
- [Addressing Ambiguity in Imitation Learning through Product of Experts based Negative Feedback](https://mlsanity.com/paper/2603.26467/) (2026)

## Abstract (arXiv)

In this study, we introduce a novel visual imitation network with a spatial attention module for robotic assisted feeding (RAF). The goal is to acquire (i.e., scoop) food items from a bowl. However, achieving robust and adaptive food manipulation is particularly challenging. To deal with this, we propose a framework that integrates visual perception with imitation learning to enable the robot to handle diverse scenarios during scooping. Our approach, named AVIL (adaptive visual imitation learning), exhibits adaptability and robustness across different bowl configurations in terms of material, size, and position, as well as diverse food types including granular, semi-solid, and liquid, even in the presence of distractors. We validate the effectiveness of our approach by conducting experiments on a real robot. We also compare its performance with a baseline. The results demonstrate improvement over the baseline across all scenarios, with an enhancement of up to 2.5 times in terms of a success metric. Notably, our model, trained solely on data from a transparent glass bowl containing granular cereals, showcases generalization ability when tested zero-shot on other bowl configurations with different types of food.