# Visual News: Benchmark and Challenges in News Image Captioning

arXiv: https://arxiv.org/abs/2010.03743 · HTML: https://mlsanity.com/paper/2010.03743/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 30 of 121)

- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [UniIR: Training and Benchmarking Universal Multimodal Information Retrievers](https://mlsanity.com/paper/2311.17136/) (2023)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities](https://mlsanity.com/paper/2302.11154/) (2023)
- [Detecting and Grounding Multi-Modal Media Manipulation](https://mlsanity.com/paper/2304.02556/) (2023)
- [Evaluation and Analysis of Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2308.15126/) (2023)
- [Detecting Multimedia Generated by Large AI Models: A Survey](https://mlsanity.com/paper/2402.00045/) (2024)
- [SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection](https://mlsanity.com/paper/2403.03170/) (2024)
- [Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources](https://mlsanity.com/paper/2112.00061/) (2021)
- [MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs](https://mlsanity.com/paper/2406.08772/) (2024)
- [Detecting and Grounding Multi-Modal Media Manipulation and Beyond](https://mlsanity.com/paper/2309.14203/) (2023)
- [Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning](https://mlsanity.com/paper/2312.10160/) (2023)
- [AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics](https://mlsanity.com/paper/2304.06870/) (2023)
- [VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias](https://mlsanity.com/paper/2304.14133/) (2023)
- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs](https://mlsanity.com/paper/2403.01988/) (2024)
- [Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps](https://mlsanity.com/paper/2307.05052/) (2023)
- [WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types](https://mlsanity.com/paper/2204.06347/) (2022)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [EDIS: Entity-Driven Image Search over Multimodal Web Content](https://mlsanity.com/paper/2305.13631/) (2023)
- [I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision](https://mlsanity.com/paper/2211.09778/) (2022)
- [MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms](https://mlsanity.com/paper/2402.14154/) (2024)
- [Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval](https://mlsanity.com/paper/2505.19650/) (2025)
- [Synthetic Misinformers: Generating and Combating Multimodal Misinformation](https://mlsanity.com/paper/2303.01217/) (2023)
- [Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying](https://mlsanity.com/paper/2506.02020/) (2025)
- [Rethinking Image Editing Detection in the Era of Generative AI Revolution](https://mlsanity.com/paper/2311.17953/) (2023)

## Abstract (arXiv)

We propose Visual News Captioner, an entity-aware model for the task of news image captioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. Unlike the standard image captioning task, news images depict situations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and textual features to generate captions with richer information such as events and entities. More specifically, built upon the Transformer architecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms, which are designed to generate named entities more accurately. Our method utilizes much fewer parameters while achieving slightly better prediction results than competing methods. Our larger and more diverse Visual News dataset further highlights the remaining challenges in captioning news images.