# From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning

arXiv: https://arxiv.org/abs/1708.02478 · HTML: https://mlsanity.com/paper/1708.02478/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2017.

## Built on by (most cited first, 15 of 15)

- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [Structured Two-stream Attention Network for Video Question Answering](https://mlsanity.com/paper/2206.01017/) (2022)
- [One-shot Scene Graph Generation](https://mlsanity.com/paper/2202.10824/) (2022)
- [SEM-POS: Grammatically and Semantically Correct Video Captioning](https://mlsanity.com/paper/2303.14829/) (2023)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty](https://mlsanity.com/paper/2210.13729/) (2022)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [NeuroView-RNN: It's About Time](https://mlsanity.com/paper/2202.11811/) (2022)
- [The Role of the Input in Natural Language Video Description](https://mlsanity.com/paper/2102.05067/) (2021)
- [FlowNar: Scalable Streaming Narration for Long-Form Videos](https://mlsanity.com/paper/2606.00620/) (2026)
- [How to Understand Named Entities: Using Common Sense for News Captioning](https://mlsanity.com/paper/2403.06520/) (2024)
- [EVC-MF: End-to-end Video Captioning Network with Multi-scale Features](https://mlsanity.com/paper/2410.16624/) (2024)
- [A Statistical Framework for Model Selection in LSTM Networks](https://mlsanity.com/paper/2506.06840/) (2025)

## Abstract (arXiv)

Video captioning in essential is a complex natural process, which is affected by various uncertainties stemming from video content, subjective judgment, etc. In this paper we build on the recent progress in using encoder-decoder framework for video captioning and address what we find to be a critical deficiency of the existing methods, that most of the decoders propagate deterministic hidden states. Such complex uncertainty cannot be modeled efficiently by the deterministic models. In this paper, we propose a generative approach, referred to as multi-modal stochastic RNNs networks (MS-RNN), which models the uncertainty observed in the data using latent stochastic variables. Therefore, MS-RNN can improve the performance of video captioning, and generate multiple sentences to describe a video considering different random factors. Specifically, a multi-modal LSTM (M-LSTM) is first proposed to interact with both visual and textual features to capture a high-level representation. Then, a backward stochastic LSTM (S-LSTM) is proposed to support uncertainty propagation by introducing latent variables. Experimental results on the challenging datasets MSVD and MSR-VTT show that our proposed MS-RNN approach outperforms the state-of-the-art video captioning benchmarks.