mlsanityData to 14 Oct 2026

Housekeep: Tidying Virtual Households using Commonsense Reasoning

arXiv 2205.10712 ↗First posted 22 May 2022cs.CVcited by 57

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce Housekeep, a benchmark to evaluate commonsense reasoning in the home for embodied AI. In Housekeep, an embodied agent must tidy a house by rearranging misplaced objects without explicit instructions specifying which objects need to be rearranged. Instead, the agent must learn from and is evaluated against human preferences of which objects belong where in a tidy house. Specifically, we collect a dataset of where humans typically place objects in tidy and untidy houses constituting 1799 objects, 268 object categories, 585 placements, and 105 rooms. Next, we propose a modular baseline approach for Housekeep that integrates planning, exploration, and navigation. It leverages a fine-tuned large language model (LLM) trained on an internet text corpus for effective planning. We show that our baseline agent generalizes to rearranging unseen objects in unknown environments. See our webpage for more details: https://yashkant.github.io/housekeep/

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 85 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

60 of 85 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
38 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  4. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  6. 2018On Evaluation of Embodied Navigation Agentscited by 547
  7. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  8. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  10. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  11. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  13. 2017Embodied Question Answeringcited by 336
  14. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  15. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  16. 2019Abductive Commonsense Reasoningcited by 301
  17. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  18. 2019Vision-and-Dialog Navigationcited by 250
  19. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  20. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  21. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  22. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  23. 2020How Context Affects Language Models' Factual Predictionscited by 181
  24. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  25. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  26. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  27. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  28. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  29. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 155
  30. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 123
  31. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  32. 2021Visual Room Rearrangementcited by 102
  33. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  34. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  35. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 66
  36. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  37. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  38. 2020An Exploration of Embodied Visual Explorationcited by 58

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 21; 2024: 16; 2025: 9; 2026: 7.

57 papers.

Sort:

By area:

  1. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  2. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  4. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  5. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  6. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  7. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  8. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 57
  9. 2022Retrospectives on the Embodied AI Workshopcited by 46
  10. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  11. 2024HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environmentscited by 32
  12. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 29
38 more
  1. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  2. 2024Materials science in the era of large language models: a perspectivecited by 24
  3. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  4. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
  5. 2023Out of the Cage: How Stochastic Parrots Win in Cyber Security Environmentscited by 18
  6. 2024LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robotscited by 18
  7. 2023Pre-Trained Large Language Models for Industrial Controlcited by 16
  8. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  9. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  10. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  11. 2022TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specificationcited by 13
  12. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  13. 2022Robot Task Planning and Situation Handling in Open Worldscited by 10
  14. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  15. 2024"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situationscited by 9
  16. 2024"Set It Up!": Functional Object Arrangement with Compositional Generative Modelscited by 9
  17. 2024Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environmentscited by 9
  18. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  19. 2023LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusorcited by 7
  20. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 7
  21. 2023Heterogeneous Embodied Multi-Agent Collaborationcited by 6
  22. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  23. 2023LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Queryingcited by 5
  24. 2023SceneScore: Learning a Cost Function for Object Arrangementcited by 4
  25. 2023LLM-SAP: Large Language Models Situational Awareness Based Planningcited by 4
  26. 2024Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Frameworkcited by 4
  27. 2025LEGO-SLAM: Language-Embedded Gaussian Optimization SLAMcited by 3
  28. 2026How VLAs (Really) Work In Open-World Environmentscited by 3
  29. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  30. 2023A Reminder of its Brittleness: Language Reward Shaping May Hinder Learning for Instruction Following Agentscited by 2
  31. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  32. 2025P3: Toward Versatile Embodied Agentscited by 2
  33. 2026From Instruction to Event: Sound-Triggered Mobile Manipulationcited by 2
  34. 2026Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planningcited by 2
  35. 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agentscited by 2
  36. 2023Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environmentscited by 1
  37. 2023Multimodal Contextualized Plan Prediction for Embodied Task Completioncited by 1
  38. 2025Personalized Robotic Object Rearrangement from Scene Contextcited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Physical Reasoning and Object Planning for Household Embodied Agents
  2. 2026TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning
  3. 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
  4. 2026Neurosymbolic Embodied Agents
  5. 2025HELP: Hierarchical Embodied Language Planner for Household Tasks
  6. 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
  7. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
  8. 2026MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks