mlsanityData to 7 Oct 2026

Imitating Task and Motion Planning with Visuomotor Transformers

arXiv 2305.16309 ↗First posted 25 May 2023cs.RO cs.CV cs.LGcited by 63

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning is a powerful tool for training robot manipulation policies, allowing them to learn from expert demonstrations without manual programming or trial-and-error. However, common methods of data collection, such as human supervision, scale poorly, as they are time-consuming and labor-intensive. In contrast, Task and Motion Planning (TAMP) can autonomously generate large-scale datasets of diverse demonstrations. In this work, we show that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for robot manipulation. To that end, we present a novel imitation learning system called OPTIMUS that trains large-scale visuomotor Transformer policies by imitating a TAMP agent. OPTIMUS introduces a pipeline for generating TAMP data that is specifically curated for imitation learning and can be used to train performant transformer-based policies. In this paper, we present a thorough study of the design decisions required to imitate TAMP and demonstrate that OPTIMUS can solve a wide variety of challenging vision-based manipulation tasks with over 70 different objects, ranging from long-horizon pick-and-place tasks, to shelf and articulated object manipulation, achieving 70 to 80% success rates. Video results and code at https://mihdalal.github.io/optimus/

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 77 of its 103 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

77 of 103 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learning19 Sep
  2. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  4. 2022Practical Imitation Learning in the Real World via Task Consistency Loss3 Feb
  5. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation9 Dec
  6. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learning31 Mar
  7. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  8. 2020Learning Multi-Arm Manipulation Through Collaborative Teleoperation12 Dec
  9. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperation12 Dec
  10. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  11. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov
  12. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills26 Feb

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2023Segment Anythingcited by 8,158
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2022A Generalist Agentcited by 802
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  14. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  15. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  16. 2020Reinforcement Learning with Augmented Datacited by 452
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  18. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  19. 2019Learning Latent Plans from Playcited by 310
  20. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  21. 2020Integrated Task and Motion Planningcited by 281
  22. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  23. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  24. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  25. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  26. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  27. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 198
  28. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  29. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  30. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  31. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  32. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  33. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  34. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  35. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  36. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  37. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  38. 2018Motion Planning Networkscited by 78

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 18; 2025: 23; 2026: 13.

63 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  3. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  4. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  6. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  7. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  8. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  9. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  10. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  11. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  12. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
38 more
  1. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  2. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  3. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  4. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  5. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  6. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  7. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  8. 2024Potential Based Diffusion Motion Planningcited by 40
  9. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  10. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  11. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  12. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 24
  13. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 21
  14. 2025What Matters in Learning from Large-Scale Datasets for Robot Manipulationcited by 20
  15. 2024Neural MP: A Generalist Neural Motion Plannercited by 18
  16. 2024GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitationcited by 14
  17. 2024LEGENT: Open Platform for Embodied Agentscited by 12
  18. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 12
  19. 2025MTIL: Encoding Full History with Mamba for Temporal Imitation Learningcited by 10
  20. 2024InCoRo: In-Context Learning for Robotics Control with Feedback Loopscited by 9
  21. 2023NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptorscited by 8
  22. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulationcited by 8
  23. 2025Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformerscited by 8
  24. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  25. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  26. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  27. 2025VLA Model-Expert Collaboration for Bi-directional Manipulation Learningcited by 5
  28. 2023STAMP: Differentiable Task and Motion Planning via Stein Variational Gradient Descentcited by 4
  29. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  30. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  31. 2026Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Roboticscited by 4
  32. 2025Is an object-centric representation beneficial for robotic manipulation ?cited by 3
  33. 2025SLAP: Shortcut Learning for Abstract Planningcited by 3
  34. 2025ReinforceGen: Hybrid Skill Policies with Automated Data Generation and Reinforcement Learningcited by 3
  35. 2026MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalizationcited by 3
  36. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  37. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  38. 2026FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulationcited by 2

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations ↗
  2. 2025SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives
  3. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  4. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learning
  5. 2025Latent Diffusion Planning for Imitation Learning
  6. 2020Visual Imitation Made Easy
  7. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training
  8. 2025Masked Generative Policy for Robotic Control