mlsanityData to 7 Oct 2026

VirtualHome: Simulating Household Activities via Programs

arXiv 1806.07011 ↗First posted 19 Jun 2018cs.CV cs.AI cs.LGcited by 411

Not on the map. Why?

Cited by 95 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.

Key facts. Cited by 411 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 36; 2022: 47; 2023: 71; 2024: 89; 2025: 101; 2026: 66.

411 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  9. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
38 more
  1. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  2. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  7. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  8. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  9. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  10. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  11. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  12. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  13. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  14. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  15. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  16. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  17. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  18. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  19. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  20. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  21. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  22. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  23. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  24. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  25. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  26. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  27. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  28. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  29. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  30. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  31. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  32. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  33. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  34. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  35. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  36. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  37. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  38. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100

50 most cited of 411 All 411 (JSON)