# Entity-aware Image Caption Generation

arXiv: https://arxiv.org/abs/1804.07889 · HTML: https://mlsanity.com/paper/1804.07889/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2018.

## Built on by (most cited first, 26 of 26)

- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [A Decade of Knowledge Graphs in Natural Language Processing: A Survey](https://mlsanity.com/paper/2210.00105/) (2022)
- [Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph](https://mlsanity.com/paper/2107.11970/) (2021)
- [Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation](https://mlsanity.com/paper/2109.05743/) (2021)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [ICECAP: Information Concentrated Entity-aware Image Captioning](https://mlsanity.com/paper/2108.02050/) (2021)
- [CapOnImage: Context-driven Dense-Captioning on Image](https://mlsanity.com/paper/2204.12974/) (2022)
- [Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model](https://mlsanity.com/paper/2308.01126/) (2023)
- [IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System](https://mlsanity.com/paper/2109.04202/) (2021)
- [Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction](https://mlsanity.com/paper/2104.08874/) (2021)
- [Redefining "Hallucination" in LLMs: Towards a psychology-informed framework for mitigating misinformation](https://mlsanity.com/paper/2402.01769/) (2024)
- [Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach](https://mlsanity.com/paper/2410.23676/) (2024)
- [IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images](https://mlsanity.com/paper/2305.10438/) (2023)
- [Generating image captions with external encyclopedic knowledge](https://mlsanity.com/paper/2210.04806/) (2022)
- [Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2403.06735/) (2024)
- [EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning](https://mlsanity.com/paper/2402.19404/) (2024)
- [A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)](https://mlsanity.com/paper/2406.19054/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation](https://mlsanity.com/paper/2504.18269/) (2025)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)
- [Towards Multimodal Vision-Language Models Generating Non-Generic Text](https://mlsanity.com/paper/2207.04174/) (2022)
- [HICL: Hashtag-Driven In-Context Learning for Social Media Natural Language Understanding](https://mlsanity.com/paper/2308.09985/) (2023)
- [Video Summarization: Towards Entity-Aware Captions](https://mlsanity.com/paper/2312.02188/) (2023)
- [TROPE: TRaining-Free Object-Part Enhancement for Seamlessly Improving Fine-Grained Zero-Shot Image Captioning](https://mlsanity.com/paper/2409.19960/) (2024)
- [A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entities](https://mlsanity.com/paper/2410.13803/) (2024)
- [Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model](https://mlsanity.com/paper/2505.23358/) (2025)

## Abstract (arXiv)

Current image captioning approaches generate descriptions which lack specific information, such as named entities that are involved in the images. In this paper we propose a new task which aims to generate informative image captions, given images and hashtags as input. We propose a simple but effective approach to tackle this problem. We first train a convolutional neural networks - long short term memory networks (CNN-LSTM) model to generate a template caption based on the input image. Then we use a knowledge graph based collective inference algorithm to fill in the template with specific named entities retrieved via the hashtags. Experiments on a new benchmark dataset collected from Flickr show that our model generates news-style image descriptions with much richer information. Our model outperforms unimodal baselines significantly with various evaluation metrics.