# Deep soccer captioning with transformer: dataset, semantics-related losses, and multi-level evaluation

arXiv: https://arxiv.org/abs/2202.05728 · HTML: https://mlsanity.com/paper/2202.05728/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos](https://mlsanity.com/paper/2011.13367/) (2020)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Grounded Video Description](https://mlsanity.com/paper/1812.06587/) (2018)
- [Compositional Generalization in Image Captioning](https://mlsanity.com/paper/1909.04402/) (2019)
- [Improved Soccer Action Spotting using both Audio and Video Streams](https://mlsanity.com/paper/2011.04258/) (2020)
- [RMS-Net: Regression and Masking for Soccer Event Spotting](https://mlsanity.com/paper/2102.07624/) (2021)
- [A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions](https://mlsanity.com/paper/1910.02930/) (2019)
- [An empirical study on the effectiveness of images in Multimodal Neural Machine Translation](https://mlsanity.com/paper/1707.00995/) (2017)
- [Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation](https://mlsanity.com/paper/1703.08084/) (2017)
- [Multi-level Attention Fusion Network for Audio-visual Event Recognition](https://mlsanity.com/paper/2106.06736/) (2021)
- [Modulating and attending the source image during encoding improves Multimodal Translation](https://mlsanity.com/paper/1712.03449/) (2017)
- [UMONS Submission for WMT18 Multimodal Translation Task](https://mlsanity.com/paper/1810.06233/) (2018)
- [AVECL-UMONS database for audio-visual event classification and localization](https://mlsanity.com/paper/2011.01018/) (2020)
- [Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation](https://mlsanity.com/paper/1707.01009/) (2017)

## Built on by (most cited first, 4 of 4)

- [SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries](https://mlsanity.com/paper/2304.04565/) (2023)
- [Player-Centric Multimodal Prompt Generation for Large Language Model Based Identity-Aware Basketball Video Captioning](https://mlsanity.com/paper/2507.20163/) (2025)
- [Shot2Tactic-Caption: Multi-Scale Captioning of Badminton Videos for Tactical Understanding](https://mlsanity.com/paper/2510.14617/) (2025)
- [Sentinel2Cap: A Human-Annotated Benchmark Dataset for Multimodal Remote Sensing Image Captioning](https://mlsanity.com/paper/2605.03189/) (2026)

## Abstract (arXiv)

This work aims at generating captions for soccer videos using deep learning. In this context, this paper introduces a dataset, model, and triple-level evaluation. The dataset consists of 22k caption-clip pairs and three visual features (images, optical flow, inpainting) for ~500 hours of SoccerNet videos. The model is divided into three parts: a transformer learns language, ConvNets learn vision, and a fusion of linguistic and visual features generates captions. The paper suggests evaluating generated captions at three levels: syntax (the commonly used evaluation metrics such as BLEU-score and CIDEr), meaning (the quality of descriptions for a domain expert), and corpus (the diversity of generated captions). The paper shows that the diversity of generated captions has improved (from 0.07 reaching 0.18) with semantics-related losses that prioritize selected words. Semantics-related losses and the utilization of more visual features (optical flow, inpainting) improved the normalized captioning score by 28%. The web page of this work: https://sites.google.com/view/soccercaptioninghttps://sites.google.com/view/soccercaptioning