mlsanityData to 7 Oct 2026

Transformers are Adaptable Task Planners

arXiv 2207.02442 ↗First posted 6 Jul 2022cs.RO cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Every home is different, and every person likes things done in their particular way. Therefore, home robots of the future need to both reason about the sequential nature of day-to-day tasks and generalize to user's preferences. To this end, we propose a Transformer Task Planner(TTP) that learns high-level actions from demonstrations by leveraging object attribute-based representations. TTP can be pre-trained on multiple preferences and shows generalization to unseen preferences using a single demonstration as a prompt in a simulated dishwasher loading task. Further, we demonstrate real-world dish rearrangement using TTP with a Franka Panda robotic arm, prompted using a single human demonstration.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 60 of its 74 references that are in the map. First posted on arXiv 6 Jul 2022.

Builds on

60 of 74 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Mask R-CNNcited by 7,259
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  3. 2020Object-Centric Learning with Slot Attentioncited by 865
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  11. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  12. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  13. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  14. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  15. 2020Integrated Task and Motion Planningcited by 281
  16. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  17. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  18. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  19. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  20. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  21. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  22. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  23. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  24. 2017Intel RealSense Stereoscopic Depth Camerascited by 82
  25. 2016Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challengecited by 70
  26. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  27. 2020Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentationcited by 64
  28. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  29. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  30. 2022Models of human preference for learning reward functionscited by 54
  31. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  32. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  33. 2021PlaTe: Visually-Grounded Planning with Transformers in Procedural Taskscited by 45
  34. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  35. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  36. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  37. 2021Differentiable Spatial Planning using Transformerscited by 25
  38. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 4; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  2. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  3. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  4. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  5. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  6. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  7. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  8. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  9. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  10. 2023SceneScore: Learning a Cost Function for Object Arrangementcited by 4
  11. 2023LLM-SAP: Large Language Models Situational Awareness Based Planningcited by 4
  12. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
3 more
  1. 2023Knolling Bot: Teaching Robots the Human Notion of Tidinesscited by 0
  2. 2024PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangementcited by 0
  3. 2026Offline Decision Transformers for Neural Combinatorial Optimization: Surpassing Heuristics on the Traveling Salesman Problemcited by 0