# Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution

arXiv: https://arxiv.org/abs/2412.13492 · HTML: https://mlsanity.com/paper/2412.13492/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Real-World Humanoid Locomotion with Reinforcement Learning](https://mlsanity.com/paper/2303.03381/) (2023)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Language Instructed Reinforcement Learning for Human-AI Coordination](https://mlsanity.com/paper/2304.07297/) (2023)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [Thinking inside the box: A tutorial on grey-box Bayesian optimization](https://mlsanity.com/paper/2201.00272/) (2022)

## Built on by (most cited first, 5 of 5)

- [RHO: Your Coding Agent is Secretly a Roboticist](https://mlsanity.com/paper/2606.16458/) (2026)
- [ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents](https://mlsanity.com/paper/2606.21262/) (2026)
- [Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Design](https://mlsanity.com/paper/2608.10299/) (2026)
- [CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning](https://mlsanity.com/paper/2607.01721/) (2026)
- [A Bird's-Eye View of Iterative Reward Design](https://mlsanity.com/paper/2610.04364/) (2026)

## Abstract (arXiv)

The ability to autonomously explore and resolve tasks with minimal human guidance is crucial for the self-development of embodied intelligence. Although reinforcement learning methods can largely ease human effort, it's challenging to design reward functions for real-world tasks, especially for high-dimensional robotic control, due to complex relationships among joints and tasks. Recent advancements large language models (LLMs) enable automatic reward function design. However, approaches evaluate reward functions by re-training policies from scratch placing an undue burden on the reward function, expecting it to be effective throughout the whole policy improvement process. We argue for a more practical strategy in robotic autonomy, focusing on refining existing policies with policy-dependent reward functions rather than a universal one. To this end, we propose a novel reward-policy co-evolution framework where the reward function and the learned policy benefit from each other's progressive on-the-fly improvements, resulting in more efficient and higher-performing skill acquisition. Specifically, the reward evolution process translates the robot's previous best reward function, descriptions of tasks and environment into text inputs. These inputs are used to query LLMs to generate a dynamic amount of reward function candidates, ensuring continuous improvement at each round of evolution. For policy evolution, our method generates new policy populations by hybridizing historically optimal and random policies. Through an improved Bayesian optimization, our approach efficiently and robustly identifies the most capable and plastic reward-policy combination, which then proceeds to the next round of co-evolution. Despite using less data, our approach demonstrates an average normalized improvement of 95.3% across various high-dimensional robotic skill learning tasks.