mlsanityData to 14 Oct 2026

Skill Transformer: A Monolithic Policy for Mobile Manipulation

arXiv 2308.09873 ↗First posted 19 Aug 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We present Skill Transformer, an approach for solving long-horizon robotic tasks by combining conditional sequence modeling and skill modularity. Conditioned on egocentric and proprioceptive observations of a robot, Skill Transformer is trained end-to-end to predict both a high-level skill (e.g., navigation, picking, placing), and a whole-body low-level action (e.g., base and arm motion), using a transformer architecture and demonstration trajectories that solve the full task. It retains the composability and modularity of the overall task through a skill predictor module while reasoning about low-level actions and avoiding hand-off errors, common in modular approaches. We test Skill Transformer on an embodied rearrangement benchmark and find it performs robust task planning and low-level control in new scenarios, achieving a 2.5x higher success rate than baselines in hard rearrangement problems.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  7. 2022A Generalist Agentcited by 808
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
26 more
  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  2. 2021Implicit Behavioral Cloningcited by 452
  3. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  5. 2020Integrated Task and Motion Planningcited by 290
  6. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  7. 2022Multi-Game Decision Transformerscited by 214
  8. 2022Online Decision Transformercited by 196
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  10. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  11. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  12. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  13. 2021Visual Room Rearrangementcited by 102
  14. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  15. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  16. 2019HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulatorscited by 55
  17. 2017Multi-Level Discovery of Deep Optionscited by 51
  18. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
  19. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  20. 2021Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulationcited by 44
  21. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  22. 2021Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learningcited by 26
  23. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  24. 2022VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangementcited by 18
  25. 2019Whole-Body MPC for a Dynamically Stable Mobile Manipulatorcited by 13
  26. 2022Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformerscited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 8; 2025: 6.

16 papers.

Sort:

By area:

  1. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  2. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  3. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  4. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  5. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  6. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  7. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  8. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  9. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
  10. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  11. 2025SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulationcited by 9
  12. 2024M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenescited by 6
4 more
  1. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  2. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  3. 2023Unlock Reliable Skill Inference for Quadruped Adaptive Behavior by Skill Graphcited by 0
  4. 2025An Uncertainty-Weighted Decision Transformer for Navigation in Dense, Complex Driving Scenarioscited by 0