mlsanityData to 14 Oct 2026

MetaMorph: Learning Universal Controllers with Transformers

arXiv 2203.11931 ↗First posted 22 Mar 2022cs.LG cs.NE cs.ROcited by 89

Not on the map. Why?

Abstract (arXiv)

Multiple domains like vision, natural language, and audio are witnessing tremendous progress by leveraging Transformers for large scale pre-training followed by task specific fine tuning. In contrast, in robotics we primarily train a single robot for a single task. However, modular robot systems now allow for the flexible combination of general-purpose building blocks into task optimized morphologies. However, given the exponentially large number of possible robot morphologies, training a controller for each new design is impractical. In this work, we propose MetaMorph, a Transformer based approach to learn a universal controller over a modular robot design space. MetaMorph is based on the insight that robot morphology is just another modality on which we can condition the output of a Transformer. Through extensive experiments we demonstrate that large scale pre-training on a variety of robot morphologies results in policies with combinatorial generalization capabilities, including zero shot generalization to unseen robot morphologies. We further demonstrate that our pre-trained policy can be used for sample-efficient transfer to completely new robot morphologies and tasks.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

30 of 52 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Layer Normalizationcited by 5,669
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  9. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
18 more
  1. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  2. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  3. 2018Graph networks as learnable physics engines for inference and controlcited by 244
  4. 2015Semi-supervised Sequence Learningcited by 198
  5. 2021Embodied Intelligence via Learning and Evolutioncited by 163
  6. 2020One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Controlcited by 121
  7. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  8. 2019Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularitycited by 70
  9. 2020My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Controlcited by 70
  10. 2018Reinforcement Learning for Improving Agent Designcited by 56
  11. 2018Jointly Learning to Construct and Control Agents using Deep Reinforcement Learningcited by 55
  12. 2018Hardware Conditioned Policies for Multi-Robot Transfer Learningcited by 46
  13. 2017Scalable Co-Optimization of Morphology and Control in Embodied Machinescited by 40
  14. 2019Data-efficient Learning of Morphology and Controller for a Microrobotcited by 26
  15. 2021Task-Agnostic Morphology Evolutioncited by 24
  16. 2021Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platformscited by 22
  17. 2020Policy Transfer via Kinematic Domain Randomization and Adaptationcited by 16
  18. 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 12; 2024: 22; 2025: 23; 2026: 23.

89 papers.

Sort:

By area:

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  7. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  8. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  9. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  10. 2024One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotioncited by 43
  11. 2024Body Transformer: Leveraging Robot Embodiment for Policy Learningcited by 32
  12. 2023Learning to Modulate pre-trained Models in RLcited by 28
38 more
  1. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  2. 2023A Generalist Dynamics Model for Controlcited by 24
  3. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understandingcited by 24
  4. 2023Universal Morphology Control via Contextual Modulationcited by 23
  5. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  6. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20
  7. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  8. 2022Low-Rank Modular Reinforcement Learning via Muscle Synergycited by 16
  9. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  10. 2025Towards Embodiment Scaling Laws in Robot Locomotioncited by 16
  11. 2024ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learningcited by 15
  12. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 13
  13. 2023Symmetry-Aware Robot Design with Structured Subgroupscited by 13
  14. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  15. 2023Subequivariant Graph Reinforcement Learning in 3D Environmentscited by 11
  16. 2024Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Controlcited by 11
  17. 2025UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policiescited by 11
  18. 2025Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objectscited by 10
  19. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  20. 2025BodyGen: Advancing Towards Efficient Embodiment Co-Designcited by 9
  21. 2025GCNT: Graph-Based Transformer Policies for Morphology-Agnostic Reinforcement Learningcited by 7
  22. 2024DittoGym: Learning to Control Soft Shape-Shifting Robotscited by 6
  23. 2025Accelerated co-design of robots through morphological pretrainingcited by 6
  24. 2025AnyBody: A Benchmark Suite for Cross-Embodiment Manipulationcited by 6
  25. 2023Curriculum Reinforcement Learning via Morphology-Environment Co-Evolutioncited by 5
  26. 2023Leveraging Hyperbolic Embeddings for Coarse-to-Fine Robot Designcited by 5
  27. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  28. 2022Minimal Neural Network Models for Permutation Invariant Agentscited by 4
  29. 2023Feudal Graph Reinforcement Learningcited by 4
  30. 2023Autonomy 2.0: The Quest for Economies of Scalecited by 4
  31. 2024Task2Morph: Differentiable Task-inspired Framework for Contact-Aware Robot Designcited by 4
  32. 2024HeteroMorpheus: Universal Control Based on Morphological Heterogeneity Modelingcited by 4
  33. 2024Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Controlcited by 4
  34. 2025Trajectory World Models for Heterogeneous Environmentscited by 4
  35. 2025Multi-Embodiment Locomotion at Scale with extreme Embodiment Randomizationcited by 4
  36. 2025Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learningcited by 4
  37. 2025Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challengescited by 4
  38. 2026Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotioncited by 4

50 most cited of 89 All 89 (JSON)