How to Understand Named Entities: Using Common Sense for News Captioning
Not on the map. Why?
Abstract (arXiv)
News captioning aims to describe an image with its news article body as input. It greatly relies on a set of detected named entities, including real-world people, organizations, and places. This paper exploits commonsense knowledge to understand named entities for news captioning. By ``understand'', we mean correlating the news content with common sense in the wild, which helps an agent to 1) distinguish semantically similar named entities and 2) describe named entities using words outside of training corpora. Our approach consists of three modules: (a) Filter Module aims to clarify the common sense concerning a named entity from two aspects: what does it mean? and what is it related to?, which divide the common sense into explanatory knowledge and relevant knowledge, respectively. (b) Distinguish Module aggregates explanatory knowledge from node-degree, dependency, and distinguish three aspects to distinguish semantically similar named entities. (c) Enrich Module attaches relevant knowledge to named entities to enrich the entity description by commonsense information (e.g., identity and social position). Finally, the probability distributions from both modules are integrated to generate the news captions. Extensive experiments on two challenging datasets (i.e., GoodNews and NYTimes) demonstrate the superiority of our method. Ablation studies and visualization further validate its effectiveness in understanding named entities.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 56 of its 96 references that are in the map. First posted on arXiv 11 Mar 2024.
Builds on
56 of 96 references.
References · 56
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
- 2018YOLOv3: An Incremental Improvementcited by 2,659
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
38 more
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 903
- 2014Show and Tell: A Neural Image Caption Generatorcited by 879
- 2016Self-critical Sequence Training for Image Captioningcited by 547
- 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 388
- 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
- 2019Attention on Attention for Image Captioningcited by 289
- 2016Image Captioning with Semantic Attentioncited by 260
- 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
- 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
- 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
- 2020X-Linear Attention Networks for Image Captioningcited by 198
- 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
- 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
- 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
- 2020FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platformscited by 78
- 2019Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answeringcited by 68
- 2016Efficient softmax approximation for GPUscited by 67
- 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66
- 2020Normalized and Geometry-Aware Self-Attention Network for Image Captioningcited by 63
- 2020Transform and Tell: Entity-Aware News Image Captioningcited by 52
- 2018Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledgecited by 48
- 2019Leveraging Knowledge Bases in LSTMs for Improving Machine Readingcited by 37
- 2020Length-Controllable Image Captioningcited by 36
- 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
- 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
- 2016BreakingNews: Article Annotation by Image and Text Processingcited by 20
- 2023APoLLo: Unified Adapter and Prompt Learning for Vision Language Modelscited by 19
- 2018Hierarchical LSTMs with Adaptive Attention for Visual Captioningcited by 18
- 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
- 2017From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioningcited by 15
- 2018Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processingcited by 15
- 2023LAMM: Label Alignment for Multi-Modal Prompt Learningcited by 15
- 2021Journalistic Guidelines Aware News Image Captioningcited by 14
- 2023Improving Image Captioning via Predicting Structured Conceptscited by 13
50 most cited of 56 All 56 (JSON)
Built on by
Not cited yet.