# Hardware as Policy: Mechanical and Computational Co-Optimization using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.04460 · HTML: https://mlsanity.com/paper/2008.04460/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2020.

## Built on by (most cited first, 30 of 32)

- [An End-to-End Differentiable Framework for Contact-Aware Robot Design](https://mlsanity.com/paper/2107.07501/) (2021)
- [Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design](https://mlsanity.com/paper/2110.03659/) (2021)
- [Task-Agnostic Morphology Evolution](https://mlsanity.com/paper/2102.13100/) (2021)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures](https://mlsanity.com/paper/2504.17857/) (2025)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design](https://mlsanity.com/paper/2402.15038/) (2024)
- [MORPH: Design Co-optimization with Reinforcement Learning via a Differentiable Hardware Model Proxy](https://mlsanity.com/paper/2309.17227/) (2023)
- [Text2Robot: Evolutionary Robot Design from Text Descriptions](https://mlsanity.com/paper/2406.19963/) (2024)
- [Learning to Design and Use Tools for Robotic Manipulation](https://mlsanity.com/paper/2311.00754/) (2023)
- [Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution](https://mlsanity.com/paper/2309.12529/) (2023)
- [Task2Morph: Differentiable Task-inspired Framework for Contact-Aware Robot Design](https://mlsanity.com/paper/2403.19093/) (2024)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Reinforcement Learning for Efficient Design and Control Co-optimisation of Energy Systems](https://mlsanity.com/paper/2406.19825/) (2024)
- [What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks](https://mlsanity.com/paper/2111.02371/) (2021)
- [Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks](https://mlsanity.com/paper/2202.09275/) (2022)
- [C^2:Co-design of Robots via Concurrent Networks Coupling Online and Offline Reinforcement Learning](https://mlsanity.com/paper/2209.06579/) (2022)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Latent Diffeomorphic Co-Design of End-Effectors for Deformable and Fragile Object Manipulation](https://mlsanity.com/paper/2602.17921/) (2026)
- [Robust In-Hand Manipulation via Priors in Reinforcement Learning and Mechanical Design](https://mlsanity.com/paper/2607.12105/) (2026)
- [BRIDGE: An Open-Source Humanoid Platform via Morphology-Control Co-Design for Physical AI](https://mlsanity.com/paper/2609.03497/) (2026)
- [Unified Simulation, Perception, and Generation of Human Behavior](https://mlsanity.com/paper/2204.13678/) (2022)
- [Selecting Mechanical Parameters of a Monopode Jumping System with Reinforcement Learning](https://mlsanity.com/paper/2212.01303/) (2022)
- [Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception](https://mlsanity.com/paper/2506.20045/) (2025)
- [Digital Twin-enabled Multi-generation Control Co-Design with Deep Reinforcement Learning](https://mlsanity.com/paper/2510.10694/) (2025)
- [Digital Twin-based Control Co-Design of Full Vehicle Active Suspensions via Deep Reinforcement Learning](https://mlsanity.com/paper/2512.03891/) (2025)
- [RAPID: Reconfigurable, Adaptive Platform for Iterative Design](https://mlsanity.com/paper/2602.06653/) (2026)
- [A Reinforcement Learning-based Transmission Expansion Framework Considering Strategic Bidding in Electricity Markets](https://mlsanity.com/paper/2602.19421/) (2026)
- [Sizing of Battery Considering Renewable Energy Bidding Strategy with Reinforcement Learning](https://mlsanity.com/paper/2602.19428/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (RL) has shown great success in learning complex control policies for a variety of applications in robotics. However, in most such cases, the hardware of the robot has been considered immutable, modeled as part of the environment. In this study, we explore the problem of learning hardware and control parameters together in a unified RL framework. To achieve this, we propose to model the robot body as a "hardware policy", analogous to and optimized jointly with its computational counterpart. We show that, by modeling such hardware policies as auto-differentiable computational graphs, the ensuing optimization problem can be solved efficiently by gradient-based algorithms from the Policy Optimization family. We present two such design examples: a toy mass-spring problem, and a real-world problem of designing an underactuated hand. We compare our method against traditional co-optimization approaches, and also demonstrate its effectiveness by building a physical prototype based on the learned hardware parameters. Videos and more details are available at https://roamlab.github.io/hwasp/ .