# R2D-RL: A RoboCup 2D Soccer Environment for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2606.18786 · HTML: https://mlsanity.com/paper/2606.18786/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space](https://mlsanity.com/paper/1810.06394/) (2018)
- [Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning](https://mlsanity.com/paper/2305.17886/) (2023)
- [Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations](https://mlsanity.com/paper/2305.13030/) (2023)
- [Pyrus Base: An Open Source Python Framework for the RoboCup 2D Soccer Simulation](https://mlsanity.com/paper/2307.16875/) (2023)
- [Cross Language Soccer Framework: An Open Source Framework for the RoboCup 2D Soccer Simulation](https://mlsanity.com/paper/2406.05621/) (2024)

## Abstract (arXiv)

Robot soccer is a challenging testbed for multi-agent reinforcement learning because it combines partial observability, cooperative and adversarial interaction, sparse rewards, and long-horizon tactical behavior. RoboCup 2D Soccer Simulation (RCSS2D) provides a mature robot-soccer platform, but its competition-oriented server-client architecture is difficult to use directly with modern Python-based MARL workflows. We introduce R2D-RL, a reinforcement learning environment that connects RCSS2D and HELIOS-based player clients to a Python MARL interface through shared-memory communication and cycle-level synchronization. R2D-RL supports full-field and scenario-based training with configurable opponents, Base discrete and Hybrid parameterized action spaces, action masks, expected possession value (EPV)-based reward shaping, and parallel execution. We provide front-goal scenarios and an 11-vs-11 full-field benchmark, together with baseline results.