mlsanityData to 7 Oct 2026

Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

arXiv 2010.09890 ↗First posted 19 Oct 2020cs.AI cs.LG cs.MAcited by 131

Not on the map. Why?

Cited by 34 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of AI agents with the human-like agent as well as with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that the proposed challenge and virtual environment enable a systematic evaluation on the important aspects of machine social intelligence at scale.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 16; 2023: 19; 2024: 34; 2025: 29; 2026: 25.

131 papers.

Sort:

By area:

  1. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  2. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  3. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  4. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  5. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  6. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  7. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
  8. 2023MindAgent: Emergent Gaming Interactioncited by 87
  9. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  10. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
38 more
  1. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 65
  2. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  3. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 57
  4. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  5. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  6. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  7. 2022Retrospectives on the Embodied AI Workshopcited by 46
  8. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 43
  9. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  10. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 41
  11. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  12. 2023Learning Human-to-Robot Handovers from Point Cloudscited by 34
  13. 2023Interactive Natural Language Processingcited by 31
  14. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  15. 2024Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planningcited by 28
  16. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  17. 2022Last-Mile Embodied Visual Navigationcited by 25
  18. 2021A Survey on Human-aware Robot Navigationcited by 23
  19. 2023NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistantscited by 23
  20. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  21. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 20
  22. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 19
  23. 2022Proactive Robot Assistance via Spatio-Temporal Object Modelingcited by 18
  24. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  25. 2024GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignmentcited by 16
  26. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 16
  27. 2026Learning Personalized Agents from Human Feedbackcited by 16
  28. 2022HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handoverscited by 15
  29. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  30. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  31. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  32. 2022Concept Graph Neural Networks for Surgical Video Understandingcited by 13
  33. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  34. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  35. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  36. 2024Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Householdscited by 11
  37. 2024ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Modelscited by 11
  38. 2025LLM-Based Social Simulations Require a Boundarycited by 11

50 most cited of 131 All 131 (JSON)