# SentiCap: Generating Image Descriptions with Sentiments

arXiv: https://arxiv.org/abs/1510.01431 · HTML: https://mlsanity.com/paper/1510.01431/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2015.

## Built on by (most cited first, 30 of 48)

- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Caption Anything: Interactive Image Description with Diverse Multimodal Controls](https://mlsanity.com/paper/2305.02677/) (2023)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features](https://mlsanity.com/paper/2207.09666/) (2022)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought](https://mlsanity.com/paper/2401.06836/) (2024)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision](https://mlsanity.com/paper/2211.09778/) (2022)
- [Style-Aware Contrastive Learning for Multi-Style Image Captioning](https://mlsanity.com/paper/2301.11367/) (2023)
- [Controllable Image Captioning via Prompting](https://mlsanity.com/paper/2212.01803/) (2022)
- [Dual-path Collaborative Generation Network for Emotional Video Captioning](https://mlsanity.com/paper/2408.03006/) (2024)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Learning Combinatorial Prompts for Universal Controllable Image Captioning](https://mlsanity.com/paper/2303.06338/) (2023)
- [Affective Image Editing: Shaping Emotional Factors via Text Descriptions](https://mlsanity.com/paper/2505.18699/) (2025)
- [IterIS: Iterative Inference-Solving Alignment for LoRA Merging](https://mlsanity.com/paper/2411.15231/) (2024)
- [A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes](https://mlsanity.com/paper/2403.07469/) (2024)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets](https://mlsanity.com/paper/2111.12727/) (2021)
- [DRAG: Director-Generator Language Modelling Framework for Non-Parallel Author Stylized Rewriting](https://mlsanity.com/paper/2101.11836/) (2021)
- [Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences](https://mlsanity.com/paper/2307.16399/) (2023)
- [Training A Small Emotional Vision Language Model for Visual Art Comprehension](https://mlsanity.com/paper/2403.11150/) (2024)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)
- [Diverse Image Captioning with Grounded Style](https://mlsanity.com/paper/2205.01813/) (2022)
- [Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning](https://mlsanity.com/paper/2310.19995/) (2023)

## Abstract (arXiv)

The recent progress on image recognition and language modeling is making automatic description of image content a reality. However, stylized, non-factual aspects of the written description are missing from the current systems. One such style is descriptions with emotions, which is commonplace in everyday communication, and influences decision-making and interpersonal relationships. We design a system to describe an image with emotions, and present a model that automatically generates captions with positive or negative sentiments. We propose a novel switching recurrent neural network with word-level regularization, which is able to produce emotional image captions using only 2000+ training sentences containing sentiments. We evaluate the captions with different automatic and crowd-sourcing metrics. Our model compares favourably in common quality metrics for image captioning. In 84.6% of cases the generated positive captions were judged as being at least as descriptive as the factual captions. Of these positive captions 88% were confirmed by the crowd-sourced workers as having the appropriate sentiment.