# Tune to Learn: How Controller Gains Shape Robot Policy Learning

arXiv: https://arxiv.org/abs/2604.02523 · HTML: https://mlsanity.com/paper/2604.02523/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [skrl: Modular and Flexible Library for Reinforcement Learning](https://mlsanity.com/paper/2202.03825/) (2022)
- [On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer](https://mlsanity.com/paper/2312.03673/) (2023)
- [Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfer](https://mlsanity.com/paper/2304.09434/) (2023)
- [SAIL: Faster-than-Demonstration Execution of Imitation Learning Policies](https://mlsanity.com/paper/2506.11948/) (2025)
- [SoftMimic: Learning Compliant Whole-body Control from Examples](https://mlsanity.com/paper/2510.17792/) (2025)
- [Learning Low-Frequency Motion Control for Robust and Dynamic Robot Locomotion](https://mlsanity.com/paper/2209.14887/) (2022)
- [cmaes: A Simple yet Practical Python Library for CMA-ES](https://mlsanity.com/paper/2402.01373/) (2024)
- [DexHub and DART: Towards Internet Scale Robot Data Collection](https://mlsanity.com/paper/2411.02214/) (2024)
- [Automatic Environment Shaping is the Next Frontier in RL](https://mlsanity.com/paper/2407.16186/) (2024)

## Built on by (most cited first, 8 of 8)

- [TempoVLA: Learning Speed-Controllable Vision-Language-Action Policies](https://mlsanity.com/paper/2606.06491/) (2026)
- [TacCoRL: Integrating Tactile Feedback into VLA via Simulation](https://mlsanity.com/paper/2606.11743/) (2026)
- [CLIFT: Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuning](https://mlsanity.com/paper/2607.29172/) (2026)
- [Behavior Cloning Under PD Control: A Finite-Horizon Theory of Gain-Dependent Error Amplification](https://mlsanity.com/paper/2604.14484/) (2026)
- [SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies](https://mlsanity.com/paper/2606.24049/) (2026)
- [TaskNPoint: How to Teach Your Humanoid to Hit a Backhand in Minutes](https://mlsanity.com/paper/2606.26215/) (2026)
- [Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies](https://mlsanity.com/paper/2606.29898/) (2026)
- [Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning](https://mlsanity.com/paper/2609.09492/) (2026)

## Abstract (arXiv)

Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn