# Learning Modular Robot Control Policies

arXiv: https://arxiv.org/abs/2105.10049 · HTML: https://mlsanity.com/paper/2105.10049/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 90 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [Learning Stable Deep Dynamics Models](https://mlsanity.com/paper/2001.06116/) (2020)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving](https://mlsanity.com/paper/1707.02342/) (2017)
- [GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning](https://mlsanity.com/paper/1810.05762/) (2018)
- [Deep Drone Acrobatics](https://mlsanity.com/paper/2006.05768/) (2020)
- [A Game Theoretic Framework for Model Based Reinforcement Learning](https://mlsanity.com/paper/2004.07804/) (2020)
- [Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity](https://mlsanity.com/paper/1902.05546/) (2019)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning](https://mlsanity.com/paper/1801.01432/) (2018)
- [Scalable Co-Optimization of Morphology and Control in Embodied Machines](https://mlsanity.com/paper/1706.06133/) (2017)
- [Data-efficient Co-Adaptation of Morphology and Behaviour with Deep Reinforcement Learning](https://mlsanity.com/paper/1911.06832/) (2019)
- [An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Research](https://mlsanity.com/paper/1910.00093/) (2019)
- [Combining Physical Simulators and Object-Based Networks for Control](https://mlsanity.com/paper/1904.06580/) (2019)

## Built on by (most cited first, 22 of 22)

- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [ManyQuadrupeds: Learning a Single Locomotion Policy for Diverse Quadruped Robots](https://mlsanity.com/paper/2310.10486/) (2023)
- [REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer](https://mlsanity.com/paper/2202.05244/) (2022)
- [BodyGen: Advancing Towards Efficient Embodiment Co-Design](https://mlsanity.com/paper/2503.00533/) (2025)
- [DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies](https://mlsanity.com/paper/2310.05714/) (2023)
- [DMAP: a Distributed Morphological Attention Policy for Learning to Locomote with a Changing Body](https://mlsanity.com/paper/2209.14218/) (2022)
- [COMPOSER: Scalable and Robust Modular Policies for Snake Robots](https://mlsanity.com/paper/2310.00871/) (2023)
- [Decentralized Motor Skill Learning for Complex Robotic Systems](https://mlsanity.com/paper/2306.17411/) (2023)
- [Simulation-Aided Policy Tuning for Black-Box Robot Learning](https://mlsanity.com/paper/2411.14246/) (2024)
- [Beyond Topology: A Morphological Symmetry Graph Representation for Locomotion Policy Learning](https://mlsanity.com/paper/2512.00727/) (2025)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotion](https://mlsanity.com/paper/2604.08780/) (2026)
- [Embedding Morphology into Transformers for Cross-Robot Policy Learning](https://mlsanity.com/paper/2603.00182/) (2026)
- [Metarobotics for Industry and Society: Vision, Technologies, and Opportunities](https://mlsanity.com/paper/2404.00797/) (2024)
- [Toward Humanoid Brain-Body Co-design: Joint Optimization of Control and Morphology for Fall Recovery](https://mlsanity.com/paper/2510.22336/) (2025)
- [Sample Efficient Dynamics Learning for Symmetrical Legged Robots:Leveraging Physics Invariance and Geometric Symmetries](https://mlsanity.com/paper/2210.07329/) (2022)
- [MeMo: Meaningful, Modular Controllers via Noise Injection](https://mlsanity.com/paper/2407.01567/) (2024)
- [Multi-Modal Decentralized Reinforcement Learning for Modular Reconfigurable Lunar Robots](https://mlsanity.com/paper/2510.20347/) (2025)
- [Malleable Agents for Re-Configurable Robotic Manipulators](https://mlsanity.com/paper/2202.02395/) (2022)
- [Learning Modular Robot Locomotion from Demonstrations](https://mlsanity.com/paper/2210.17491/) (2022)
- [Morphology-Aware Graph Reinforcement Learning for Tensegrity Robot Locomotion](https://mlsanity.com/paper/2510.26067/) (2025)
- [Distributed Multi Robot Lunar Cargo Transportation via Phase Decomposed Reinforcement Learning](https://mlsanity.com/paper/2607.00160/) (2026)

## Abstract (arXiv)

Modular robots can be rearranged into a new design, perhaps each day, to handle a wide variety of tasks by forming a customized robot for each new task. However, reconfiguring just the mechanism is not sufficient: each design also requires its own unique control policy. One could craft a policy from scratch for each new design, but such an approach is not scalable, especially given the large number of designs that can be generated from even a small set of modules. Instead, we create a modular policy framework where the policy structure is conditioned on the hardware arrangement, and use just one training process to create a policy that controls a wide variety of designs. Our approach leverages the fact that the kinematics of a modular robot can be represented as a design graph, with nodes as modules and edges as connections between them. Given a robot, its design graph is used to create a policy graph with the same structure, where each node contains a deep neural network, and modules of the same type share knowledge via shared parameters (e.g., all legs on a hexapod share the same network parameters). We developed a model-based reinforcement learning algorithm, interleaving model learning and trajectory optimization to train the policy. We show the modular policy generalizes to a large number of designs that were not seen during training without any additional learning. Finally, we demonstrate the policy controlling a variety of designs to locomote with both simulated and real robots.