# Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control

arXiv: https://arxiv.org/abs/2509.01765 · HTML: https://mlsanity.com/paper/2509.01765/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [Adapting Auxiliary Losses Using Gradient Similarity](https://mlsanity.com/paper/1812.02224/) (2018)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions](https://mlsanity.com/paper/2203.15103/) (2022)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.09754/) (2024)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [HumanMimic: Learning Natural Locomotion and Transitions for Humanoid Robot via Wasserstein Adversarial Imitation](https://mlsanity.com/paper/2309.14225/) (2023)
- [Composite Motion Learning with Task Control](https://mlsanity.com/paper/2305.03286/) (2023)
- [Symmetry Considerations for Learning Task Symmetric Robot Policies](https://mlsanity.com/paper/2403.04359/) (2024)
- [Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking](https://mlsanity.com/paper/2404.19173/) (2024)
- [RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards](https://mlsanity.com/paper/2407.11562/) (2024)
- [Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond](https://mlsanity.com/paper/2501.10945/) (2025)
- [Learning Terrain-Adaptive Locomotion with Agile Behaviors by Imitating Animals](https://mlsanity.com/paper/2308.03273/) (2023)

## Built on by (most cited first, 5 of 5)

- [Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolution](https://mlsanity.com/paper/2509.14816/) (2025)
- [Toward Reliable Sim-to-Real Predictability for MoE-based Robust Quadrupedal Locomotion](https://mlsanity.com/paper/2602.00678/) (2026)
- [SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking](https://mlsanity.com/paper/2609.06061/) (2026)
- [D-JEPA: A Decision-Aligned Latent World Model](https://mlsanity.com/paper/2609.24749/) (2026)
- [DAMPER: Return-Prioritized Gradient Control for Smooth Policies](https://mlsanity.com/paper/2609.38903/) (2026)

## Abstract (arXiv)

Efficient robot control often requires balancing task performance with energy expenditure. A common approach in reinforcement learning (RL) is to penalize energy use directly as part of the reward function. This requires carefully tuning weight terms to avoid undesirable trade-offs where energy minimization harms task success. In this work, we propose a hyperparameter-free gradient optimization method to minimize energy expenditure without conflicting with task performance. Inspired by recent works in multitask learning, our method applies policy gradient projection between task and energy objectives to derive policy updates that minimize energy expenditure in ways that do not impact task performance. We evaluate this technique on standard locomotion benchmarks of DM-Control and HumanoidBench and demonstrate a reduction of 64% energy usage while maintaining comparable task performance. Further, we conduct experiments on a Unitree GO2 quadruped showcasing Sim2Real transfer of energy efficient policies. Our method is easy to implement in standard RL pipelines with minimal code changes, is applicable to any policy gradient method, and offers a principled alternative to reward shaping for energy efficient control policies.