# Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning

arXiv: https://arxiv.org/abs/1908.08529 · HTML: https://mlsanity.com/paper/1908.08529/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 21 of 21)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Make-A-Story: Visual Memory Conditioned Consistent Story Generation](https://mlsanity.com/paper/2211.13319/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion](https://mlsanity.com/paper/2306.11593/) (2023)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [Guiding Visual Question Generation](https://mlsanity.com/paper/2110.08226/) (2021)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Question-controlled Text-aware Image Captioning](https://mlsanity.com/paper/2108.02059/) (2021)
- [LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies](https://mlsanity.com/paper/2108.13459/) (2021)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Variational Stacked Local Attention Networks for Diverse Video Captioning](https://mlsanity.com/paper/2201.00985/) (2022)
- [Diverse Image Captioning with Grounded Style](https://mlsanity.com/paper/2205.01813/) (2022)
- [Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights](https://mlsanity.com/paper/2407.11449/) (2024)
- [CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentation](https://mlsanity.com/paper/2407.11393/) (2024)
- [ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with Unpaired Stylistic Corpora](https://mlsanity.com/paper/2308.01143/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [Image Embedding Sampling Method for Diverse Captioning](https://mlsanity.com/paper/2502.10118/) (2025)
- [Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention](https://mlsanity.com/paper/2504.02496/) (2025)

## Abstract (arXiv)

Diverse and accurate vision+language modeling is an important goal to retain creative freedom and maintain user engagement. However, adequately capturing the intricacies of diversity in language models is challenging. Recent works commonly resort to latent variable models augmented with more or less supervision from object detectors or part-of-speech tags. Common to all those methods is the fact that the latent variable either only initializes the sentence generation process or is identical across the steps of generation. Both methods offer no fine-grained control. To address this concern, we propose Seq-CVAE which learns a latent space for every word position. We encourage this temporal latent space to capture the 'intention' about how to complete the sentence by mimicking a representation which summarizes the future. We illustrate the efficacy of the proposed approach to anticipate the sentence continuation on the challenging MSCOCO dataset, significantly improving diversity metrics compared to baselines while performing on par w.r.t sentence quality.