# Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot

arXiv: https://arxiv.org/abs/2208.01160 · HTML: https://mlsanity.com/paper/2208.01160/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion](https://mlsanity.com/paper/2202.05481/) (2022)
- [Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World](https://mlsanity.com/paper/2110.05457/) (2021)
- [Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control](https://mlsanity.com/paper/1909.06586/) (2019)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [Deep Predictive Policy Training using Reinforcement Learning](https://mlsanity.com/paper/1703.00727/) (2017)
- [Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning](https://mlsanity.com/paper/1703.03078/) (2017)
- [Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs](https://mlsanity.com/paper/2011.08811/) (2020)
- [Autonomous Navigation for Quadrupedal Robots with Optimized Jumping through Constrained Obstacles](https://mlsanity.com/paper/2107.00773/) (2021)

## Built on by (most cited first, 30 of 38)

- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Expressive Whole-Body Control for Humanoid Robots](https://mlsanity.com/paper/2402.16796/) (2024)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction](https://mlsanity.com/paper/2405.10315/) (2024)
- [GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots](https://mlsanity.com/paper/2209.05309/) (2022)
- [DribbleBot: Dynamic Legged Manipulation in the Wild](https://mlsanity.com/paper/2304.01159/) (2023)
- [Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion](https://mlsanity.com/paper/2303.11330/) (2023)
- [Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models](https://mlsanity.com/paper/2308.15143/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies](https://mlsanity.com/paper/2410.11825/) (2024)
- [Pedipulate: Enabling Manipulation Skills using a Quadruped Robot's Leg](https://mlsanity.com/paper/2402.10837/) (2024)
- [Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning](https://mlsanity.com/paper/2210.04435/) (2022)
- [Learning Visual Quadrupedal Loco-Manipulation from Demonstrations](https://mlsanity.com/paper/2403.20328/) (2024)
- [Bridging the Sim-to-Real Gap for Athletic Loco-Manipulation](https://mlsanity.com/paper/2502.10894/) (2025)
- [SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending](https://mlsanity.com/paper/2506.09366/) (2025)
- [Humanoid Goalkeeper: Learning from Position Conditioned Task-Motion Constraints](https://mlsanity.com/paper/2510.18002/) (2025)
- [PA-LOCO: Learning Perturbation-Adaptive Locomotion for Quadruped Robots](https://mlsanity.com/paper/2407.04224/) (2024)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [HiLMa-Res: A General Hierarchical Framework via Residual RL for Combining Quadrupedal Locomotion and Manipulation](https://mlsanity.com/paper/2407.06584/) (2024)
- [VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Play](https://mlsanity.com/paper/2502.01932/) (2025)
- [DexDribbler: Learning Dexterous Soccer Manipulation via Dynamic Supervision](https://mlsanity.com/paper/2403.14300/) (2024)
- [Toward Real-World Cooperative and Competitive Soccer with Quadrupedal Robot Teams](https://mlsanity.com/paper/2505.13834/) (2025)
- [Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2507.20382/) (2025)
- [Facilitating Sim-to-real by Intrinsic Stochasticity of Real-Time Simulation in Reinforcement Learning for Robot Manipulation](https://mlsanity.com/paper/2304.06056/) (2023)
- [AURA: Autonomous Upskilling with Retrieval-Augmented Agents](https://mlsanity.com/paper/2506.02507/) (2025)
- [Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble](https://mlsanity.com/paper/2212.03562/) (2022)
- [Efficient Reinforcement Learning for Jumping Monopods](https://mlsanity.com/paper/2309.07038/) (2023)
- [A Multifidelity Sim-to-Real Pipeline for Verifiable and Compositional Reinforcement Learning](https://mlsanity.com/paper/2312.01249/) (2023)

## Abstract (arXiv)

We address the problem of enabling quadrupedal robots to perform precise shooting skills in the real world using reinforcement learning. Developing algorithms to enable a legged robot to shoot a soccer ball to a given target is a challenging problem that combines robot motion control and planning into one task. To solve this problem, we need to consider the dynamics limitation and motion stability during the control of a dynamic legged robot. Moreover, we need to consider motion planning to shoot the hard-to-model deformable ball rolling on the ground with uncertain friction to a desired location. In this paper, we propose a hierarchical framework that leverages deep reinforcement learning to train (a) a robust motion control policy that can track arbitrary motions and (b) a planning policy to decide the desired kicking motion to shoot a soccer ball to a target. We deploy the proposed framework on an A1 quadrupedal robot and enable it to accurately shoot the ball to random targets in the real world.