Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning
On the map: Retrieval-augmented generation → Multimodal RAG
Abstract (arXiv)
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 48 references that are in the map. First posted on arXiv 26 Nov 2025.
Builds on
25 of 48 references.
References · 25
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2017Graph Attention Networkscited by 8,923
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2017VGGFace2: A dataset for recognising faces across pose and agecited by 914
- 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
- 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
- 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
13 more
- 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
- 2021Sample and Computation Redistribution for Efficient Face Detectioncited by 89
- 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
- 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
- 2018The Devil of Face Recognition is in the Noisecited by 65
- 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
- 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
- 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
- 2016BreakingNews: Article Annotation by Image and Text Processingcited by 20
- 2021Journalistic Guidelines Aware News Image Captioningcited by 14
- 2023Visually-Aware Context Modeling for News Image Captioningcited by 12
- 2021ICECAP: Information Concentrated Entity-aware Image Captioningcited by 11
- 2022Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipediacited by 10
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning
- 2024EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
- 2026CIAN: Multi-Stage Framework for Event-Enriched Image Captioning via Retrieval-Augmented Generation
- 2024Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization
- 2025ReCap: Event-Aware Image Captioning with Article Retrieval and Semantic Gaussian Normalization
- 2025OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
- 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
- 2024Towards Retrieval-Augmented Architectures for Image Captioning