# "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention

arXiv: https://arxiv.org/abs/1807.03871 · HTML: https://mlsanity.com/paper/1807.03871/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2018.

## Built on by (most cited first, 26 of 26)

- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [Style-Aware Contrastive Learning for Multi-Style Image Captioning](https://mlsanity.com/paper/2301.11367/) (2023)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [A Circular-Structured Representation for Visual Emotion Distribution Learning](https://mlsanity.com/paper/2106.12450/) (2021)
- [Controllable Image Captioning via Prompting](https://mlsanity.com/paper/2212.01803/) (2022)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Learning Combinatorial Prompts for Universal Controllable Image Captioning](https://mlsanity.com/paper/2303.06338/) (2023)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets](https://mlsanity.com/paper/2111.12727/) (2021)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences](https://mlsanity.com/paper/2307.16399/) (2023)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)
- [3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model](https://mlsanity.com/paper/2103.11186/) (2021)
- [BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance](https://mlsanity.com/paper/2506.03589/) (2025)
- [ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with Unpaired Stylistic Corpora](https://mlsanity.com/paper/2308.01143/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Semi-supervised Chinese Poem-to-Painting Generation via Cycle-consistent Adversarial Networks](https://mlsanity.com/paper/2410.19307/) (2024)
- [Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning](https://mlsanity.com/paper/2607.29045/) (2026)
- [A Self-Explainable Stylish Image Captioning Framework via Multi-References](https://mlsanity.com/paper/2110.10704/) (2021)
- [StyleM: Stylized Metrics for Image Captioning Built with Contrastive N-grams](https://mlsanity.com/paper/2201.00975/) (2022)
- [Probing the Limits of Stylistic Alignment in Vision-Language Models](https://mlsanity.com/paper/2509.25568/) (2025)
- [RoadTones: Tone Controllable Text Generation from Road Event Videos](https://mlsanity.com/paper/2605.21411/) (2026)

## Abstract (arXiv)

Generating stylized captions for an image is an emerging topic in image captioning. Given an image as input, it requires the system to generate a caption that has a specific style (e.g., humorous, romantic, positive, and negative) while describing the image content semantically accurately. In this paper, we propose a novel stylized image captioning model that effectively takes both requirements into consideration. To this end, we first devise a new variant of LSTM, named style-factual LSTM, as the building block of our model. It uses two groups of matrices to capture the factual and stylized knowledge, respectively, and automatically learns the word-level weights of the two groups based on previous context. In addition, when we train the model to capture stylized elements, we propose an adaptive learning approach based on a reference factual model, it provides factual knowledge to the model as the model learns from stylized caption labels, and can adaptively compute how much information to supply at each time step. We evaluate our model on two stylized image captioning datasets, which contain humorous/romantic captions and positive/negative captions, respectively. Experiments shows that our proposed model outperforms the state-of-the-art approaches, without using extra ground truth supervision.