# Towards optimized actions in critical situations of soccer games with deep reinforcement learning

arXiv: https://arxiv.org/abs/2109.06625 · HTML: https://mlsanity.com/paper/2109.06625/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Actions Speak Louder Than Goals: Valuing Player Actions in Soccer](https://mlsanity.com/paper/1802.07127/) (2018)
- [Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluation](https://mlsanity.com/paper/1805.11088/) (2018)
- [SoccerMap: A Deep Learning Architecture for Visually-Interpretable Analysis in Soccer](https://mlsanity.com/paper/2010.10202/) (2020)
- [Cracking the Black Box: Distilling Deep Sports Analytics](https://mlsanity.com/paper/2006.04551/) (2020)
- [QPass: a Merit-based Evaluation of Soccer Passes](https://mlsanity.com/paper/1608.03532/) (2016)

## Built on by (most cited first, 1 of 1)

- [OpenSTARLab: Open Approach for Spatio-Temporal Agent Data Analysis in Soccer](https://mlsanity.com/paper/2502.02785/) (2025)

## Abstract (arXiv)

Soccer is a sparse rewarding game: any smart or careless action in critical situations can change the result of the match. Therefore players, coaches, and scouts are all curious about the best action to be performed in critical situations, such as the times with a high probability of losing ball possession or scoring a goal. This work proposes a new state representation for the soccer game and a batch reinforcement learning to train a smart policy network. This network gets the contextual information of the situation and proposes the optimal action to maximize the expected goal for the team. We performed extensive numerical experiments on the soccer logs made by InStat for 104 European soccer matches. The results show that in all 104 games, the optimized policy obtains higher rewards than its counterpart in the behavior policy. Besides, our framework learns policies that are close to the expected behavior in the real world. For instance, in the optimized policy, we observe that some actions such as foul, or ball out can be sometimes more rewarding than a shot in specific situations.