# Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

arXiv: https://arxiv.org/abs/2603.02655 · HTML: https://mlsanity.com/paper/2603.02655/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 8 of 8)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [Simultaneous Machine Translation with Large Language Models](https://mlsanity.com/paper/2309.06706/) (2023)
- [SCBench: A Sports Commentary Benchmark for Video LLMs](https://mlsanity.com/paper/2412.17637/) (2024)
- [Simultaneous Neural Machine Translation with Constituent Label Prediction](https://mlsanity.com/paper/2110.13480/) (2021)
- [Commentary Generation from Data Records of Multiplayer Strategy Esports Game](https://mlsanity.com/paper/2212.10935/) (2022)

## Built on by (most cited first, 1 of 1)

- [Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation](https://mlsanity.com/paper/2606.13322/) (2026)

## Abstract (arXiv)

Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.