# Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer

arXiv: https://arxiv.org/abs/2412.09417 · HTML: https://mlsanity.com/paper/2412.09417/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics](https://mlsanity.com/paper/2407.10648/) (2024)
- [Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real](https://mlsanity.com/paper/1908.05224/) (2019)
- [Learning Vision-Based Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2309.14594/) (2023)
- [DribbleBot: Dynamic Legged Manipulation in the Wild](https://mlsanity.com/paper/2304.01159/) (2023)
- [Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot](https://mlsanity.com/paper/2208.01160/) (2022)
- [Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation](https://mlsanity.com/paper/2207.10821/) (2022)
- [Learning Memory-Based Control for Human-Scale Bipedal Locomotion](https://mlsanity.com/paper/2006.02402/) (2020)
- [Achieving Human Level Competitive Robot Table Tennis](https://mlsanity.com/paper/2408.03906/) (2024)
- [Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning](https://mlsanity.com/paper/2210.04435/) (2022)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations](https://mlsanity.com/paper/2110.04507/) (2021)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1909.12324/) (2019)
- [Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion](https://mlsanity.com/paper/2008.11867/) (2020)
- [Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped](https://mlsanity.com/paper/1809.10811/) (2018)
- [Multi-fidelity reinforcement learning framework for shape optimization](https://mlsanity.com/paper/2202.11170/) (2022)
- [Designing a skilled soccer team for RoboCup: exploring skill-set-primitives through reinforcement learning](https://mlsanity.com/paper/2312.14360/) (2023)
- [Multi-Fidelity Reinforcement Learning for Time-Optimal Quadrotor Re-planning](https://mlsanity.com/paper/2403.08152/) (2024)
- [Black-Box Safety Validation of Autonomous Systems: A Multi-Fidelity Reinforcement Learning Approach](https://mlsanity.com/paper/2203.03451/) (2022)

## Built on by (most cited first, 7 of 7)

- [Toward Real-World Cooperative and Competitive Soccer with Quadrupedal Robot Teams](https://mlsanity.com/paper/2505.13834/) (2025)
- [Closed-Loop Action Chunks with Dynamic Corrections for Training-Free Diffusion Policy](https://mlsanity.com/paper/2603.01953/) (2026)
- [GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning](https://mlsanity.com/paper/2601.20753/) (2026)
- [Multi-Robot Collaboration through Reinforcement Learning and Abstract Simulation](https://mlsanity.com/paper/2503.05092/) (2025)
- [Abstract Sim2Real through Approximate Information States](https://mlsanity.com/paper/2604.15289/) (2026)
- [Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots](https://mlsanity.com/paper/2604.19104/) (2026)
- [Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates](https://mlsanity.com/paper/2610.14042/) (2026)

## Abstract (arXiv)

Robot decision-making in partially observable, real-time, dynamic, and multi-agent environments remains a difficult and unsolved challenge. Model-free reinforcement learning (RL) is a promising approach to learning decision-making in such domains, however, end-to-end RL in complex environments is often intractable. To address this challenge in the RoboCup Standard Platform League (SPL) domain, we developed a novel architecture integrating RL within a classical robotics stack, while employing a multi-fidelity sim2real approach and decomposing behavior into learned sub-behaviors with heuristic selection. Our architecture led to victory in the 2024 RoboCup SPL Challenge Shield Division. In this work, we fully describe our system's architecture and empirically analyze key design decisions that contributed to its success. Our approach demonstrates how RL-based behaviors can be integrated into complete robot behavior architectures.