# A Personalized Household Assistive Robot that Learns and Creates New Breakfast Options through Human-Robot Interaction

arXiv: https://arxiv.org/abs/2307.00114 · HTML: https://mlsanity.com/paper/2307.00114/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 28 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [EEC: Learning to Encode and Regenerate Images for Continual Learning](https://mlsanity.com/paper/2101.04904/) (2021)
- [Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks](https://mlsanity.com/paper/2105.08222/) (2021)
- [What am I allowed to do here?: Online Learning of Context-Specific Norms by Pepper](https://mlsanity.com/paper/2009.05105/) (2020)
- [The Robot Household Marathon Experiment](https://mlsanity.com/paper/2011.09792/) (2020)
- [Software architecture for YOLO, a creativity-stimulating robot](https://mlsanity.com/paper/1909.10823/) (2019)
- [Don't Forget to Buy Milk: Contextually Aware Grocery Reminder Household Robot](https://mlsanity.com/paper/2207.09050/) (2022)

## Built on by (most cited first, 1 of 1)

- [Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions](https://mlsanity.com/paper/2605.26256/) (2026)

## Abstract (arXiv)

For robots to assist users with household tasks, they must first learn about the tasks from the users. Further, performing the same task every day, in the same way, can become boring for the robot's user(s), therefore, assistive robots must find creative ways to perform tasks in the household. In this paper, we present a cognitive architecture for a household assistive robot that can learn personalized breakfast options from its users and then use the learned knowledge to set up a table for breakfast. The architecture can also use the learned knowledge to create new breakfast options over a longer period of time. The proposed cognitive architecture combines state-of-the-art perceptual learning algorithms, computational implementation of cognitive models of memory encoding and learning, a task planner for picking and placing objects in the household, a graphical user interface (GUI) to interact with the user and a novel approach for creating new breakfast options using the learned knowledge. The architecture is integrated with the Fetch mobile manipulator robot and validated, as a proof-of-concept system evaluation in a large indoor environment with multiple kitchen objects. Experimental results demonstrate the effectiveness of our architecture to learn personalized breakfast options from the user and generate new breakfast options never learned by the robot.