# Focus! Relevant and Sufficient Context Selection for News Image Captioning

arXiv: https://arxiv.org/abs/2212.00843 · HTML: https://mlsanity.com/paper/2212.00843/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 1 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [Good News, Everyone! Context driven entity-aware captioning for news images](https://mlsanity.com/paper/1904.01475/) (2019)
- [OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction](https://mlsanity.com/paper/1909.13078/) (2019)
- [Transform and Tell: Entity-Aware News Image Captioning](https://mlsanity.com/paper/2004.08070/) (2020)
- [BreakingNews: Article Annotation by Image and Text Processing](https://mlsanity.com/paper/1603.07141/) (2016)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [ICECAP: Information Concentrated Entity-aware Image Captioning](https://mlsanity.com/paper/2108.02050/) (2021)

## Built on by (most cited first, 10 of 10)

- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Visually-Aware Context Modeling for News Image Captioning](https://mlsanity.com/paper/2308.08325/) (2023)
- [RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models](https://mlsanity.com/paper/2404.04929/) (2024)
- [Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights](https://mlsanity.com/paper/2407.11449/) (2024)
- [EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning](https://mlsanity.com/paper/2402.19404/) (2024)
- [Rule-driven News Captioning](https://mlsanity.com/paper/2403.05101/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Video Summarization: Towards Entity-Aware Captions](https://mlsanity.com/paper/2312.02188/) (2023)
- [How to Understand Named Entities: Using Common Sense for News Captioning](https://mlsanity.com/paper/2403.06520/) (2024)
- [ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis](https://mlsanity.com/paper/2404.10141/) (2024)

## Abstract (arXiv)

News Image Captioning requires describing an image by leveraging additional context from a news article. Previous works only coarsely leverage the article to extract the necessary context, which makes it challenging for models to identify relevant events and named entities. In our paper, we first demonstrate that by combining more fine-grained context that captures the key named entities (obtained via an oracle) and the global context that summarizes the news, we can dramatically improve the model's ability to generate accurate news captions. This begs the question, how to automatically extract such key entities from an image? We propose to use the pre-trained vision and language retrieval model CLIP to localize the visually grounded entities in the news article and then capture the non-visual entities via an open relation extraction model. Our experiments demonstrate that by simply selecting a better context from the article, we can significantly improve the performance of existing models and achieve new state-of-the-art performance on multiple benchmarks.