# Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

arXiv: https://arxiv.org/abs/1904.01356 · HTML: https://mlsanity.com/paper/1904.01356/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 15 of 15)

- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2205.02357/) (2022)
- [RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER](https://mlsanity.com/paper/2102.02967/) (2021)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Grounding](https://mlsanity.com/paper/2211.14739/) (2022)
- [Continual Multimodal Knowledge Graph Construction](https://mlsanity.com/paper/2305.08698/) (2023)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios](https://mlsanity.com/paper/2409.17864/) (2024)
- [Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction](https://mlsanity.com/paper/2205.03521/) (2022)
- [Multimodal Question Answering for Unified Information Extraction](https://mlsanity.com/paper/2310.03017/) (2023)
- [Semantically Grounded Visual Embeddings for Zero-Shot Learning](https://mlsanity.com/paper/2201.00577/) (2022)
- [Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion](https://mlsanity.com/paper/2306.16991/) (2023)
- [Guiding Attention using Partial-Order Relationships for Image Captioning](https://mlsanity.com/paper/2204.07476/) (2022)
- [Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media](https://mlsanity.com/paper/2210.14163/) (2022)
- [Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities](https://mlsanity.com/paper/2407.16243/) (2024)

## Abstract (arXiv)

With massive explosion of social media such as Twitter and Instagram, people daily share billions of multimedia posts, containing images and text. Typically, text in these posts is short, informal and noisy, leading to ambiguities which can be resolved using images. In this paper we explore text-centric Named Entity Recognition task on these multimedia posts. We propose an end to end model which learns a joint representation of a text and an image. Our model extends multi-dimensional self attention technique, where now image helps to enhance relationship between words. Experiments show that our model is capable of capturing both textual and visual contexts with greater accuracy, achieving state-of-the-art results on Twitter multimodal Named Entity Recognition dataset.