# Reinforcement Learning for Improving Agent Design

arXiv: https://arxiv.org/abs/1810.03779 · HTML: https://mlsanity.com/paper/1810.03779/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2018.

## Built on by (most cited first, 30 of 56)

- [Embodied Intelligence via Learning and Evolution](https://mlsanity.com/paper/2102.02202/) (2021)
- [MetaMorph: Learning Universal Controllers with Transformers](https://mlsanity.com/paper/2203.11931/) (2022)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots](https://mlsanity.com/paper/2201.09863/) (2022)
- [Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design](https://mlsanity.com/paper/2110.03659/) (2021)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Task-Agnostic Morphology Evolution](https://mlsanity.com/paper/2102.13100/) (2021)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments](https://mlsanity.com/paper/2303.09555/) (2023)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer](https://mlsanity.com/paper/2202.05244/) (2022)
- [Meta Reinforcement Learning for Optimal Design of Legged Robots](https://mlsanity.com/paper/2210.02750/) (2022)
- [Symmetry-Aware Robot Design with Structured Subgroups](https://mlsanity.com/paper/2306.00036/) (2023)
- [Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networks](https://mlsanity.com/paper/2206.02887/) (2022)
- [From Universal Humanoid Control to Automatic Physically Valid Character Creation](https://mlsanity.com/paper/2206.09286/) (2022)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation](https://mlsanity.com/paper/2104.00837/) (2021)
- [EvoTorch: Scalable Evolutionary Computation in Python](https://mlsanity.com/paper/2302.12600/) (2023)
- [BodyGen: Advancing Towards Efficient Embodiment Co-Design](https://mlsanity.com/paper/2503.00533/) (2025)
- [Off-Policy Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.01138/) (2020)
- [MORPH: Design Co-optimization with Reinforcement Learning via a Differentiable Hardware Model Proxy](https://mlsanity.com/paper/2309.17227/) (2023)
- [Reinforcement learning for freeform robot design](https://mlsanity.com/paper/2310.05670/) (2023)
- [Learning to Design and Use Tools for Robotic Manipulation](https://mlsanity.com/paper/2311.00754/) (2023)
- [DittoGym: Learning to Control Soft Shape-Shifting Robots](https://mlsanity.com/paper/2401.13231/) (2024)
- [Efficient Hyperparameter Optimization for Physics-based Character Animation](https://mlsanity.com/paper/2104.12365/) (2021)
- [RoboMoRe: LLM-based Robot Co-design via Joint Optimization of Morphology and Reward](https://mlsanity.com/paper/2506.00276/) (2025)
- [Creative Problem Solving in Artificially Intelligent Agents: A Survey and Framework](https://mlsanity.com/paper/2204.10358/) (2022)

## Abstract (arXiv)

In many reinforcement learning tasks, the goal is to learn a policy to manipulate an agent, whose design is fixed, to maximize some notion of cumulative reward. The design of the agent's physical structure is rarely optimized for the task at hand. In this work, we explore the possibility of learning a version of the agent's design that is better suited for its task, jointly with the policy. We propose an alteration to the popular OpenAI Gym framework, where we parameterize parts of an environment, and allow an agent to jointly learn to modify these environment parameters along with its policy. We demonstrate that an agent can learn a better structure of its body that is not only better suited for the task, but also facilitates policy learning. Joint learning of policy and structure may even uncover design principles that are useful for assisted-design applications. Videos of results at https://designrl.github.io/