# Reinforcement learning for freeform robot design

arXiv: https://arxiv.org/abs/2310.05670 · HTML: https://mlsanity.com/paper/2310.05670/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Embodied Intelligence via Learning and Evolution](https://mlsanity.com/paper/2102.02202/) (2021)
- [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020)
- [Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity](https://mlsanity.com/paper/1902.05546/) (2019)
- [Reinforcement Learning for Improving Agent Design](https://mlsanity.com/paper/1810.03779/) (2018)
- [Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning](https://mlsanity.com/paper/1801.01432/) (2018)
- [Scalable Co-Optimization of Morphology and Control in Embodied Machines](https://mlsanity.com/paper/1706.06133/) (2017)
- [Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design](https://mlsanity.com/paper/2110.03659/) (2021)
- [Task-Agnostic Morphology Evolution](https://mlsanity.com/paper/2102.13100/) (2021)
- [Efficient automatic design of robots](https://mlsanity.com/paper/2306.03263/) (2023)
- [Functional Co-Optimization of Articulated Robots](https://mlsanity.com/paper/1707.06617/) (2017)
- [Scalable sim-to-real transfer of soft robot designs](https://mlsanity.com/paper/1911.10290/) (2019)
- [Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer](https://mlsanity.com/paper/2202.04575/) (2022)
- [4D topology optimization: Integrated optimization of the structure and self-actuation of soft bodies for dynamic motions](https://mlsanity.com/paper/2302.00905/) (2023)

## Built on by (most cited first, 8 of 8)

- [RoboMorph: Evolving Robot Morphology using Large Language Models](https://mlsanity.com/paper/2407.08626/) (2024)
- [Evolution and learning in differentiable robots](https://mlsanity.com/paper/2405.14712/) (2024)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization](https://mlsanity.com/paper/2603.15388/) (2026)
- [Toward Humanoid Brain-Body Co-design: Joint Optimization of Control and Morphology for Fall Recovery](https://mlsanity.com/paper/2510.22336/) (2025)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Efficient and Diverse Generative Robot Designs using Evolution and Intrinsic Motivation](https://mlsanity.com/paper/2411.18423/) (2024)
- [RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models](https://mlsanity.com/paper/2601.11801/) (2026)

## Abstract (arXiv)

Inspired by the necessity of morphological adaptation in animals, a growing body of work has attempted to expand robot training to encompass physical aspects of a robot's design. However, reinforcement learning methods capable of optimizing the 3D morphology of a robot have been restricted to reorienting or resizing the limbs of a predetermined and static topological genus. Here we show policy gradients for designing freeform robots with arbitrary external and internal structure. This is achieved through actions that deposit or remove bundles of atomic building blocks to form higher-level nonparametric macrostructures such as appendages, organs and cavities. Although results are provided for open loop control only, we discuss how this method could be adapted for closed loop control and sim2real transfer to physical machines in future.