# NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants

arXiv: https://arxiv.org/abs/2301.05223 · HTML: https://mlsanity.com/paper/2301.05223/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 12 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs](https://mlsanity.com/paper/2012.07277/) (2020)
- [Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction](https://mlsanity.com/paper/2002.08945/) (2020)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation](https://mlsanity.com/paper/2010.01114/) (2020)
- [BiTraP: Bi-directional Pedestrian Trajectory Prediction with Multi-modal Goal Estimation](https://mlsanity.com/paper/2007.14558/) (2020)
- [AGENT: A Benchmark for Core Psychological Reasoning](https://mlsanity.com/paper/2102.12321/) (2021)
- [Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming](https://mlsanity.com/paper/1706.00155/) (2017)
- [Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others](https://mlsanity.com/paper/2102.11938/) (2021)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [AvE: Assistance via Empowerment](https://mlsanity.com/paper/2006.14796/) (2020)
- [Goal-driven Long-Term Trajectory Prediction](https://mlsanity.com/paper/2011.02751/) (2020)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Planning With Uncertain Specifications (PUnS)](https://mlsanity.com/paper/1906.03218/) (2019)
- [Goal Inference Improves Objective and Perceived Performance in Human-Robot Collaboration](https://mlsanity.com/paper/1802.01780/) (2018)

## Built on by (most cited first, 23 of 23)

- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [COMBO: Compositional World Models for Embodied Multi-Agent Cooperation](https://mlsanity.com/paper/2404.10775/) (2024)
- [AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling](https://mlsanity.com/paper/2502.15676/) (2025)
- [MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility](https://mlsanity.com/paper/2407.08725/) (2024)
- [Towards Autonomous Micromobility through Scalable Urban Simulation](https://mlsanity.com/paper/2505.00690/) (2025)
- [Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge](https://mlsanity.com/paper/2411.01796/) (2024)
- [To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions](https://mlsanity.com/paper/2403.12533/) (2024)
- [Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind](https://mlsanity.com/paper/2409.10849/) (2024)
- [R^3-VQA: "Read the Room" by Video Social Reasoning](https://mlsanity.com/paper/2505.04147/) (2025)
- [Voluntary Collusion with Secret Tools in Competing LLM Agents](https://mlsanity.com/paper/2605.27593/) (2026)
- [Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning](https://mlsanity.com/paper/2503.12974/) (2025)
- [Visually-grounded Humanoid Agents](https://mlsanity.com/paper/2604.08509/) (2026)
- [Adaptation and Communication in Human-Robot Teaming to Handle Discrepancies in Agents' Beliefs about Plans](https://mlsanity.com/paper/2307.03362/) (2023)
- [ProToM: Promoting Prosocial Behaviour via Theory of Mind-Informed Feedback](https://mlsanity.com/paper/2509.05091/) (2025)
- [Towards Real-time Adaptation of Embodied Agent in Human-Robot Collaboration](https://mlsanity.com/paper/2412.00435/) (2024)
- [MindZero: Learning Online Mental Reasoning With Zero Annotations](https://mlsanity.com/paper/2606.00240/) (2026)
- [RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals](https://mlsanity.com/paper/2608.00289/) (2026)
- [Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots](https://mlsanity.com/paper/2609.28910/) (2026)
- [Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination](https://mlsanity.com/paper/2610.02170/) (2026)

## Abstract (arXiv)

In this work, we study how to build socially intelligent robots to assist people in their homes. In particular, we focus on assistance with online goal inference, where robots must simultaneously infer humans' goals and how to help them achieve those goals. Prior assistance methods either lack the adaptivity to adjust helping strategies (i.e., when and how to help) in response to uncertainty about goals or the scalability to conduct fast inference in a large goal space. Our NOPA (Neurally-guided Online Probabilistic Assistance) method addresses both of these challenges. NOPA consists of (1) an online goal inference module combining neural goal proposals with inverse planning and particle filtering for robust inference under uncertainty, and (2) a helping planner that discovers valuable subgoals to help with and is aware of the uncertainty in goal inference. We compare NOPA against multiple baselines in a new embodied AI assistance challenge: Online Watch-And-Help, in which a helper agent needs to simultaneously watch a main agent's action, infer its goal, and help perform a common household task faster in realistic virtual home environments. Experiments show that our helper agent robustly updates its goal inference and adapts its helping plans to the changing level of uncertainty.