# AToM-Bot: Embodied Fulfillment of Unspoken Human Needs with Affective Theory of Mind

arXiv: https://arxiv.org/abs/2406.08455 · HTML: https://mlsanity.com/paper/2406.08455/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [Modeling Others' Minds as Code](https://mlsanity.com/paper/2510.01272/) (2025)
- [MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents](https://mlsanity.com/paper/2511.23055/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations](https://mlsanity.com/paper/2609.23486/) (2026)

## Abstract (arXiv)

We propose AToM-Bot, a novel task generation and execution framework for proactive robot-human interaction, which leverages the human mental and physical state inference capabilities of the Vision Language Model (VLM) prompted by the Affective Theory of Mind (AToM). Without requiring explicit commands by humans, AToM-Bot proactively generates and follows feasible tasks to improve general human well-being. When around humans, AToM-Bot first detects current human needs based on inferred human states and observations of the surrounding environment. It then generates tasks to fulfill these needs, taking into account its embodied constraints. We designed 16 daily life scenarios spanning 4 common scenes and tasked the same visual stimulus to 59 human subjects and our robot. We used the similarity between human open-ended answers and robot output, and the human satisfaction scores to metric robot performance. AToM-Bot received high human evaluations in need detection (6.42/7, 91.7%), embodied solution (6.15/7, 87.8%) and task execution (6.17/7, 88.1%). We show that AToM-Bot excels in generating and executing feasible plans to fulfill unspoken human needs. Videos and code are available at https://affective-tom-bot.github.io.