# Proactive Robot Assistance via Spatio-Temporal Object Modeling

arXiv: https://arxiv.org/abs/2211.15501 · HTML: https://mlsanity.com/paper/2211.15501/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 46 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Representation Learning for Dynamic Graphs: A Survey](https://mlsanity.com/paper/1905.11485/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [Learning Multimodal Graph-to-Graph Translation for Molecular Optimization](https://mlsanity.com/paper/1812.01070/) (2018)
- [Molecule Edit Graph Attention Network: Modeling Chemical Reactions as Sequences of Graph Edits](https://mlsanity.com/paper/2006.15426/) (2020)
- [From Lifestyle Vlogs to Everyday Interactions](https://mlsanity.com/paper/1712.02310/) (2017)
- [Semantic Linking Maps for Active Visual Object Search](https://mlsanity.com/paper/2006.10807/) (2020)
- [Deep Multi-attributed Graph Translation with Node-Edge Co-evolution](https://mlsanity.com/paper/2003.09945/) (2020)

## Built on by (most cited first, 18 of 18)

- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling](https://mlsanity.com/paper/2502.15676/) (2025)
- [Modeling Dynamic Environments with Scene Graph Memory](https://mlsanity.com/paper/2305.17537/) (2023)
- [HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs](https://mlsanity.com/paper/2309.16524/) (2023)
- [To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions](https://mlsanity.com/paper/2403.12533/) (2024)
- [AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments](https://mlsanity.com/paper/2506.11773/) (2025)
- [3D Scene Graphs: Open Challenges and Future Directions](https://mlsanity.com/paper/2606.19383/) (2026)
- [Personalized Embodied Navigation for Portable Object Finding](https://mlsanity.com/paper/2403.09905/) (2024)
- [A Framework for Realistic Simulation of Daily Human Activity](https://mlsanity.com/paper/2311.15400/) (2023)
- [Robot Behavior Personalization from Sparse User Feedback](https://mlsanity.com/paper/2410.19219/) (2024)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [HomeEmergency -- Using Audio to Find and Respond to Emergencies in the Home](https://mlsanity.com/paper/2504.01089/) (2025)
- [Realistic Synthetic Household Data Generation at Scale](https://mlsanity.com/paper/2602.07243/) (2026)
- [Courteous Anticipation: Improving Long-Lived Task Planning in Persistent Shared Environments](https://mlsanity.com/paper/2607.20289/) (2026)
- [Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots](https://mlsanity.com/paper/2609.28910/) (2026)
- [Beyond the Remembered World: Predictive 4D Belief for Persistent Navigation in Evolving Worlds](https://mlsanity.com/paper/2609.39166/) (2026)
- [A Review Of Robotic World Models For Dynamic Environments Based On Factor And Scene Graphs](https://mlsanity.com/paper/2610.08800/) (2026)

## Abstract (arXiv)

Proactive robot assistance enables a robot to anticipate and provide for a user's needs without being explicitly asked. We formulate proactive assistance as the problem of the robot anticipating temporal patterns of object movements associated with everyday user routines, and proactively assisting the user by placing objects to adapt the environment to their needs. We introduce a generative graph neural network to learn a unified spatio-temporal predictive model of object dynamics from temporal sequences of object arrangements. We additionally contribute the Household Object Movements from Everyday Routines (HOMER) dataset, which tracks household objects associated with human activities of daily living across 50+ days for five simulated households. Our model outperforms the leading baseline in predicting object movement, correctly predicting locations for 11.1% more objects and wrongly predicting locations for 11.5% fewer objects used by the human user.