# Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generation

arXiv: https://arxiv.org/abs/1908.00249 · HTML: https://mlsanity.com/paper/1908.00249/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2019.

## Built on by (most cited first, 8 of 8)

- [Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning](https://mlsanity.com/paper/2206.01843/) (2022)
- [What Makes for Good Image Captions?](https://mlsanity.com/paper/2405.00485/) (2024)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- [HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales](https://mlsanity.com/paper/2302.12189/) (2023)
- [Enhancing image captioning with depth information using a Transformer-based framework](https://mlsanity.com/paper/2308.03767/) (2023)
- [Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning](https://mlsanity.com/paper/2105.04143/) (2021)
- [Enhanced Modality Transition for Image Captioning](https://mlsanity.com/paper/2102.11526/) (2021)
- [A Fine-Grained Image Description Generation Method Based on Joint Objectives](https://mlsanity.com/paper/2311.12799/) (2023)

## Abstract (arXiv)

Image paragraph generation is the task of producing a coherent story (usually a paragraph) that describes the visual content of an image. The problem nevertheless is not trivial especially when there are multiple descriptive and diverse gists to be considered for paragraph generation, which often happens in real images. A valid question is how to encapsulate such gists/topics that are worthy of mention from an image, and then describe the image from one topic to another but holistically with a coherent structure. In this paper, we present a new design --- Convolutional Auto-Encoding (CAE) that purely employs convolutional and deconvolutional auto-encoding framework for topic modeling on the region-level features of an image. Furthermore, we propose an architecture, namely CAE plus Long Short-Term Memory (dubbed as CAE-LSTM), that novelly integrates the learnt topics in support of paragraph generation. Technically, CAE-LSTM capitalizes on a two-level LSTM-based paragraph generation framework with attention mechanism. The paragraph-level LSTM captures the inter-sentence dependency in a paragraph, while sentence-level LSTM is to generate one sentence which is conditioned on each learnt topic. Extensive experiments are conducted on Stanford image paragraph dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, CAE-LSTM increases CIDEr performance from 20.93% to 25.15%.