# Multi-Agent Reinforcement Learning and Real-Time Decision-Making in Robotic Soccer for Virtual Environments

arXiv: https://arxiv.org/abs/2512.03166 · HTML: https://mlsanity.com/paper/2512.03166/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 41 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 5 of 5)

- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [Context-Aware Policy Reuse](https://mlsanity.com/paper/1806.03793/) (2018)
- [Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning](https://mlsanity.com/paper/2410.11022/) (2024)
- [ST-RRT*: Asymptotically-Optimal Bidirectional Motion Planning through Space-Time](https://mlsanity.com/paper/2203.02176/) (2022)
- [Starkit: RoboCup Humanoid KidSize 2021 Worldwide Champion Team Paper](https://mlsanity.com/paper/2110.08377/) (2021)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots](https://mlsanity.com/paper/2604.19104/) (2026)

## Abstract (arXiv)

The deployment of multi-agent systems in dynamic, adversarial environments like robotic soccer necessitates real-time decision-making, sophisticated cooperation, and scalable algorithms to avoid the curse of dimensionality. While Reinforcement Learning (RL) offers a promising framework, existing methods often struggle with the multi-granularity of tasks (long-term strategy vs. instant actions) and the complexity of large-scale agent interactions. This paper presents a unified Multi-Agent Reinforcement Learning (MARL) framework that addresses these challenges. First, we establish a baseline using Proximal Policy Optimization (PPO) within a client-server architecture for real-time action scheduling, with PPO demonstrating superior performance (4.32 avg. goals, 82.9% ball control). Second, we introduce a Hierarchical RL (HRL) structure based on the options framework to decompose the problem into a high-level trajectory planning layer (modeled as a Semi-Markov Decision Process) and a low-level action execution layer, improving global strategy (avg. goals increased to 5.26). Finally, to ensure scalability, we integrate mean-field theory into the HRL framework, simplifying many-agent interactions into a single agent vs. the population average. Our mean-field actor-critic method achieves a significant performance boost (5.93 avg. goals, 89.1% ball control, 92.3% passing accuracy) and enhanced training stability. Extensive simulations of 4v4 matches in the Webots environment validate our approach, demonstrating its potential for robust, scalable, and cooperative behavior in complex multi-agent domains.