# Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark

arXiv: https://arxiv.org/abs/2401.13888 · HTML: https://mlsanity.com/paper/2401.13888/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation](https://mlsanity.com/paper/2106.04632/) (2021)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries](https://mlsanity.com/paper/2304.04565/) (2023)
- [Open-book Video Captioning with Retrieve-Copy-Generate Network](https://mlsanity.com/paper/2103.05284/) (2021)
- [Social Adaptive Module for Weakly-supervised Group Activity Recognition](https://mlsanity.com/paper/2007.09470/) (2020)
- [Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning](https://mlsanity.com/paper/2003.05162/) (2020)
- [Text with Knowledge Graph Augmented Transformer for Video Captioning](https://mlsanity.com/paper/2303.12423/) (2023)
- [GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation](https://mlsanity.com/paper/2303.14655/) (2023)
- [Movie101: A New Movie Understanding Benchmark](https://mlsanity.com/paper/2305.12140/) (2023)
- [Video Storytelling: Textual Summaries for Events](https://mlsanity.com/paper/1807.09418/) (2018)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Comprehensive Information Integration Modeling Framework for Video Titling](https://mlsanity.com/paper/2006.13608/) (2020)
- [CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Matter](https://mlsanity.com/paper/2111.15162/) (2021)
- [Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval](https://mlsanity.com/paper/2309.11091/) (2023)

## Built on by (most cited first, 2 of 2)

- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [NBA_Streaming: A Large-Scale Benchmark for Fine-Grained Basketball Commentary Generation in Continuous Streams](https://mlsanity.com/paper/2608.09200/) (2026)

## Abstract (arXiv)

Despite the recent emergence of video captioning models, how to generate the text description with specific entity names and fine-grained actions is far from being solved, which however has great applications such as basketball live text broadcast. In this paper, a new multimodal knowledge graph supported basketball benchmark for video captioning is proposed. Specifically, we construct a multimodal basketball game knowledge graph (KG_NBA_2022) to provide additional knowledge beyond videos. Then, a multimodal basketball game video captioning (VC_NBA_2022) dataset that contains 9 types of fine-grained shooting events and 286 players' knowledge (i.e., images and names) is constructed based on KG_NBA_2022. We develop a knowledge guided entity-aware video captioning network (KEANet) based on a candidate player list in encoder-decoder form for basketball live text broadcast. The temporal contextual information in video is encoded by introducing the bi-directional GRU (Bi-GRU) module. And the entity-aware module is designed to model the relationships among the players and highlight the key players. Extensive experiments on multiple sports benchmarks demonstrate that KEANet effectively leverages extera knowledge and outperforms advanced video captioning models. The proposed dataset and corresponding codes will be publicly available soon