# Multimodal Memory Modelling for Video Captioning

arXiv: https://arxiv.org/abs/1611.05592 · HTML: https://mlsanity.com/paper/1611.05592/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2016.

## Built on by (most cited first, 20 of 20)

- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Relational Graph Learning for Grounded Video Description Generation](https://mlsanity.com/paper/2112.00967/) (2021)
- [Endowing Language Models with Multimodal Knowledge Graph Representations](https://mlsanity.com/paper/2206.13163/) (2022)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Boosting Video Captioning with Dynamic Loss Network](https://mlsanity.com/paper/2107.11707/) (2021)
- [Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation](https://mlsanity.com/paper/2112.14088/) (2021)
- [Co-segmentation Inspired Attention Module for Video-based Computer Vision Tasks](https://mlsanity.com/paper/2111.07370/) (2021)
- [Exploiting long-term temporal dynamics for video captioning](https://mlsanity.com/paper/2202.10828/) (2022)
- [A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video](https://mlsanity.com/paper/2312.01575/) (2023)
- [SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA](https://mlsanity.com/paper/2601.14895/) (2026)
- [Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning](https://mlsanity.com/paper/2607.29045/) (2026)
- [Global2Local: A Joint-Hierarchical Attention for Video Captioning](https://mlsanity.com/paper/2203.06663/) (2022)
- [Support-set based Multi-modal Representation Enhancement for Video Captioning](https://mlsanity.com/paper/2205.09307/) (2022)
- [DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning](https://mlsanity.com/paper/2604.08084/) (2026)

## Abstract (arXiv)

Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), video captioning has made great progress. However, learning an effective mapping from visual sequence space to language space is still a challenging problem. In this paper, we propose a Multimodal Memory Model (M3) to describe videos, which builds a visual and textual shared memory to model the long-term visual-textual dependency and further guide global visual attention on described targets. Specifically, the proposed M3 attaches an external memory to store and retrieve both visual and textual contents by interacting with video and sentence with multiple read and write operations. First, text representation in the Long Short-Term Memory (LSTM) based text decoder is written into the memory, and the memory contents will be read out to guide an attention to select related visual targets. Then, the selected visual information is written into the memory, which will be further read out to the text decoder. To evaluate the proposed model, we perform experiments on two publicly benchmark datasets: MSVD and MSR-VTT. The experimental results demonstrate that our method outperforms the state-of-theart methods in terms of BLEU and METEOR.