mlsanityData to 7 Oct 2026

TidyBot: Personalized Robot Assistance with Large Language Models

arXiv 2305.05658 ↗First posted 9 May 2023cs.RO cs.AI cs.CLcited by 219

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.

Key facts. Cited by 219 papers in the map (data through 7 Oct 2026). Builds on 48 of its 76 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

48 of 76 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
36 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  13. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  15. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  16. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  17. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  18. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  19. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  20. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  21. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  22. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  23. 2021Visual Room Rearrangementcited by 101
  24. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  25. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  26. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  27. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  28. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  29. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  30. 2022Leveraging Language for Accelerated Learning of Tool Manipulationcited by 26
  31. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  32. 2019Multi-Object Rearrangement with Monte Carlo Tree Search:A Case Study on Planar Nonprehensile Sortingcited by 12
  33. 2022Leveraging Large (Visual) Language Models for Robot 3D Scene Understandingcited by 10
  34. 2020Autonomous Planning Based on Spatial Concepts to Tidy Up Home Environments with Service Robotscited by 6
  35. 2020Decision Making in Joint Push-Grasp Action Space for Large-Scale Object Sortingcited by 5
  36. 2016Classifying and sorting cluttered piles of unknown objects with robots: a learning approachcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 56; 2024: 86; 2025: 47; 2026: 30.

219 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  7. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  8. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  9. 2023Large Language Models as General Pattern Machinescited by 196
  10. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  11. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  12. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
38 more
  1. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  2. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  3. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  4. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  6. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  7. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  10. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  11. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  12. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  13. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  14. 2023Embodied Task Planning with Large Language Modelscited by 84
  15. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  16. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  17. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 67
  18. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  19. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  20. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  21. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  22. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  23. 2024Personalized Large Language Modelscited by 44
  24. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 43
  25. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  26. 2023LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMscited by 40
  27. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  28. 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 39
  29. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  30. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  31. 2023Interactive Task Planning with Language Modelscited by 36
  32. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  33. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  34. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  35. 2024Adaptive Mobile Manipulation for Articulated Objects In the Open Worldcited by 30
  36. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  37. 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 28
  38. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28

50 most cited of 219 All 219 (JSON)