EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
Not on the map. Why?
Abstract (arXiv)
News image captioning requires model to generate an informative caption rich in entities, with the news image and the associated news article. Current MLLMs still bear limitations in handling entity information in news image captioning tasks. Besides, generating high-quality news image captions requires a trade-off between sufficiency and conciseness of textual input information. To explore the potential of MLLMs and address problems we discovered, we propose EAMA: an Entity-Aware Multimodal Alignment based approach for News Image Captioning. Our approach first aligns the MLLM with two extra alignment tasks: Entity-Aware Sentence Selection task and Entity Selection task, together with News Image Captioning task. The aligned MLLM will utilize the additional entity-related information extracted by itself to supplement the textual input while generating news image captions. Our approach achieves better results than all previous models on two mainstream news image captioning datasets.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 39 references that are in the map. First posted on arXiv 29 Feb 2024.
Builds on
20 of 39 references.
References · 20
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
- 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
- 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
- 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
- 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
- 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
8 more
- 2018Entity-aware Image Caption Generationcited by 26
- 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
- 2021Journalistic Guidelines Aware News Image Captioningcited by 14
- 2024NWPU-MOC: A Benchmark for Fine-grained Multi-category Object Counting in Aerial Imagescited by 14
- 2023Visually-Aware Context Modeling for News Image Captioningcited by 12
- 2021ICECAP: Information Concentrated Entity-aware Image Captioningcited by 11
- 2022Focus! Relevant and Sufficient Context Selection for News Image Captioningcited by 10
- 2023"Let's not Quote out of Context": Unified Vision-Language Pretraining for Context Assisted Image Captioningcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning
- 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignment
- 2024Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization
- 2026Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning
- 2024Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs
- 2026SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extraction
- 2024Semantic Alignment for Multimodal Large Language Models
- 2025Visual Semantic Description Generation with MLLMs for Image-Text Matching