# HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

arXiv: https://arxiv.org/abs/2603.00948 · HTML: https://mlsanity.com/paper/2603.00948/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids](https://mlsanity.com/paper/2502.20396/) (2025)
- [Learning and Adapting Agile Locomotion Skills by Transferring Experience](https://mlsanity.com/paper/2304.09834/) (2023)
- [Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards](https://mlsanity.com/paper/2503.11240/) (2025)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [Predictable MDP Abstraction for Unsupervised Model-Based RL](https://mlsanity.com/paper/2302.03921/) (2023)
- [Deep Reinforcement Learning for Tensegrity Robot Locomotion](https://mlsanity.com/paper/1609.09049/) (2016)
- [Designing a skilled soccer team for RoboCup: exploring skill-set-primitives through reinforcement learning](https://mlsanity.com/paper/2312.14360/) (2023)
- [Towards Equitable Agile Research and Development of AI and Robotics](https://mlsanity.com/paper/2402.08242/) (2024)
- [RobocupGym: A challenging continuous control benchmark in Robocup](https://mlsanity.com/paper/2407.14516/) (2024)
- [Mastering Agile Jumping Skills from Simple Practices with Iterative Learning Control](https://mlsanity.com/paper/2408.02619/) (2024)
- [PM-FSM: Policies Modulating Finite State Machine for Robust Quadrupedal Locomotion](https://mlsanity.com/paper/2109.12696/) (2021)

## Abstract (arXiv)

Controlling soccer robots involves multi-time-scale decision-making, which requires balancing long-term tactical planning and short-term motion execution. Traditional end-to-end reinforcement learning (RL) methods face challenges in complex dynamic environments. This paper proposes HierKick, a vision-guided soccer robot control framework based on dual-frequency hierarchical RL. The framework adopts a hierarchical control architecture featuring a 5 Hz high-level policy that integrates YOLOv8 for real-time detection and selects tasks via a coach model, and a pre-trained 50 Hz low-level controller for precise joint control. Through this architecture, the framework achieves the four steps of approaching, aligning, dribbling, and kicking. Experimental results show that the success rates of this framework are 95.2% in IsaacGym, 89.8% in Mujoco, and 80% in the real world. HierKick provides an effective hierarchical paradigm for robot control in complex environments, extendable to multi-time-scale tasks, with its modular design and skill reuse offering a new path for intelligent robot control.