# HyperPPO: A scalable method for finding small policies for robotic control

arXiv: https://arxiv.org/abs/2309.16663 · HTML: https://mlsanity.com/paper/2309.16663/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine](https://mlsanity.com/paper/2206.10558/) (2022)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [Hyper-Decision Transformer for Efficient Online Policy Adaptation](https://mlsanity.com/paper/2304.08487/) (2023)
- [Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning](https://mlsanity.com/paper/2305.13795/) (2023)
- [Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning](https://mlsanity.com/paper/2109.07735/) (2021)
- [Generating Behaviorally Diverse Policies with Latent Diffusion Models](https://mlsanity.com/paper/2305.18738/) (2023)
- [HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks](https://mlsanity.com/paper/2103.09439/) (2021)

## Built on by (most cited first, 4 of 4)

- [WARPD: World model Assisted Reactive Policy Diffusion](https://mlsanity.com/paper/2410.14040/) (2024)
- [Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworks](https://mlsanity.com/paper/2512.00085/) (2025)
- [Benefits of Low-Cost Bio-Inspiration in the Age of Overparametrization](https://mlsanity.com/paper/2604.20365/) (2026)
- [E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation](https://mlsanity.com/paper/2605.00159/) (2026)

## Abstract (arXiv)

Models with fewer parameters are necessary for the neural control of memory-limited, performant robots. Finding these smaller neural network architectures can be time-consuming. We propose HyperPPO, an on-policy reinforcement learning algorithm that utilizes graph hypernetworks to estimate the weights of multiple neural architectures simultaneously. Our method estimates weights for networks that are much smaller than those in common-use networks yet encode highly performant policies. We obtain multiple trained policies at the same time while maintaining sample efficiency and provide the user the choice of picking a network architecture that satisfies their computational constraints. We show that our method scales well - more training resources produce faster convergence to higher-performing architectures. We demonstrate that the neural policies estimated by HyperPPO are capable of decentralized control of a Crazyflie2.1 quadrotor. Website: https://sites.google.com/usc.edu/hyperppo